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FOREWORD 


Assigning  tens  of  thousands  of  Army  recruits  per  year  to  the  jobs  for  which  they  are  best 
suited  and  in  a  way  that  maximizes  aggregate  Soldier  performance  represents  a  major  goal  for 
the  Army.  The  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  has  a 
long  history  of  conducting  and  supporting  research  aimed  at  improving  the  Army’s  selection  and 
classification  process.  Effective  January  2002,  the  Army  adopted  a  set  of  nine  Aptitude  Area 
(AA)  composites,  to  select  and  classify  recruits  into  entry-level  jobs,  utilizing  weights  derived 
from  Soldier  performance  data  (circa  1989).  Relative  to  the  composites  they  replaced,  the  new 
composites  are  more  defensible  with  their  reliance  upon  actual  performance  data,  and  they  make 
use  of  the  entire  profile  of  aptitude  information  available  for  each  recruit. 

The  current  study  aimed  to  independently  evaluate  the  efficacy  of  these  new  composites 
as  well  as  alternative  AA  composites  to  meet  the  Army’s  classification  objectives.  More 
specifically,  the  present  study  tested  the  stability  and  relative  uniqueness  of  several  alternative 
AA  composite  /  job  family  structures,  and  their  practical  effects  on  classification  efficiency.  For 
both  scientific  and  practical  reasons,  the  findings  recommend  the  continued  operational  use  of 
the  nine  (standardized)  AA  composites  and  do  not  support  the  use  of  a  larger  number  of 
composites  /  job  families.  These  findings  have  been  briefed  to  the  Enlisted  Accessions  Division, 
G-l  and,  in  effect,  represent  a  pull-back  of  recommendations  based  on  earlier  research  for 
increasing  the  number  of  composites/job  families. 

MICHELLE  SAMS 
Technical  Director 
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EVALUATION  OF  ALTERNATIVE  APTITUDE  AREA  (AA)  COMPOSITES  AND  JOB 
FAMILIES  FOR  ARMY  CLASSIFICATION 


EXECUTIVE  SUMMARY _ _ 

Research  Requirement: 

To  select  and  classify  recruits  to  jobs,  the  Army  employs  nine  Aptitude  Area  (AA) 
composites.  Effective  January  2002,  the  Army  adopted  a  new  set  of  nine  AA  composites  based 
on  empirically  estimated  regression  weights.  Developed  by  Zeidner,  Johnson,  and  colleagues 
(Zeidner,  Johnson,  Vladimirsky,  &  Weldon,  2000, 2001)  with  support  from  ARI,  these 
regression-weighted  composites  were  part  of  a  proposed  two-tiered  classification  system 
designed  to  substantially  enhance  the  classification  potential  of  the  Army’s  AA  composites.  In 
an  earlier  report  (Diaz,  Ingerick,  &  Lightfoot,  2003)  we  independently  replicated  Zeidner, 
Johnson,  and  colleagues’  method  of  empirically  deriving  AA  composites,  including  the  9  AA 
composites  currently  in  operational  use.  The  primary  purpose  of  the  current  study  was  to 
evaluate  the  efficacy  of  the  proposed  AA  composites,  and  corresponding  job  families,  to  meet 
the  Army’s  classification  objectives.  Specifically,  the  present  study  tested  the  stability  and 
differential  validity  of  the  proposed  AA  composites  and  accompanying  job  families,  particularly 
the  17  and  150  relative  to  the  nine  AAs,  and  their  practical  effects  on  classification  efficiency,  as 
measured  by  mean  predicted  performance  (MPP). 

Method: 

To  assess  the  aforementioned  issues,  we  conducted  three  major  sets  of  analyses.  The  first 
set  was  exploratory  and  descriptive  and  aimed  to  evaluate  the  stability  and  reliability  of  the  9, 17, 
and  150  composites.  The  second  set  was  likewise  exploratory  in  nature  and  focused  on  assessing 
the  differential  validity  present  in  the  9  and  17  test  composites.  Specifically,  these  analyses 
tested  between-job  differences  in  both  composite  validities  and  predicted  performance  scores. 
The  third  and  final  set  of  analyses  had  two  objectives:  (1)  to  jointly  assess  the  practical  effects  of 
stability  and  differential  validity  on  MPP;  and  (2)  to  determine  the  effects  of  composite 
estimation  method  on  MPP,  and  more  operationally,  on  decisions  based  upon  MPP. 

Findings: 

Overall,  our  findings  supported  the  continued  use  of  the  standardized  AA  composites 
based  on  the  9  job  families  proposed  by  Zeidner,  Johnson,  and  colleagues,  which  are  currently  in 
operational  use  when  assigning  recruits  to  entry-level  MOS.  We  recommended  these  composites 
over  the  17  and  150  AA  composites  for  two  reasons.  First,  consistent  with  previous  research 
(Zeidner  et  al„  2003b),  moving  from  9  to  17  AA  composites  did  not  produce  a  practically 
significant  increment  in  either  overall  MPP  or  MPP  by  MOS.  Second,  and  more  importantly,  the 
9  AA  composites  based  on  standardized  weights  displayed  operationally  desirable  properties 
relative  to  unstandardized  composites.  Specifically,  standardized  composites  are  expected  to 
more  effectively  balance  the  optimization  of  aggregate  Soldier  performance  with  the  need  to 
satisfy  equally  important,  practical  requirements.  In  conclusion,  when  coupled  with  the 
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administrative  costs  and  other  management-related  issues  associated  with  changing  existing 
composites  /  job  families  and  cut  scores,  the  technical  and/or  practical  advantages  to  adopting  the 
17  or  150  AA  test  composites,  as  currently  constructed,  are  minimal. 

Use  of  Findings: 

The  present  findings,  in  conjunction  with  those  of  the  first  report  (Diaz  et  al.,  2003), 
support  the  9  AA  composites  currently  in  operational  use. 
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INTRODUCTION 


Background 


Aptitude  Areas  (AA)  are  critical  features  of  the  Army’s  personnel  management  system. 
For  more  than  50  years,  AAs  have  served  two  essential  functions.  First,  AAs  structure  and 
meaningfully  organize  entry-level  jobs,  grouping  jobs  with  similar  aptitude  requirements  into 
families.  These  families  inform  macro-level  career  field  and  accession  management  decisions, 
with  the  aim  of  ensuring  a  steady  pipeline  of  accessions  for  fulfilling  both  short-term  and 
strategic  mission  requirements.  The  second  function  served  by  AAs  is  that  each  defines  a 
tailored  composite,  which  represents  a  differentially  weighted  function  of  aptitudes  and  skills 
required  for  successful  performance  in  a  targeted  set  of  jobs.  Operationally,  these  composites 
produce  the  scores  used  to  make  a  wide  range  of  personnel  decisions,  from  setting  entry-level 
cut-scores  to  matching  new  recruits  to  jobs  to  counseling  exiting  Soldiers  on  civilian  jobs 
commensurate  with  their  skills  an>  nlities. 

First  developed  in  1949,  AAs  have  since  undergone  periodic  changes  in  number  and 
content.  From  the  early  1970s  until  recently,  the  Army  employed  a  system  of  nine  AAs.  At  the 
core  of  this  system  were  unit-weighted  composites,  whose  weights  (0, 1)  were  meant  to  reflect 
the  relative  importance  of  different  cognitive  aptitudes  and  abilities  (e.g.,  verbal  ability,  coding 
speed,  mechanical  comprehension)  in  determining  job  performance  within  a  family  of  jobs  (e.g., 
Clerical,  Combat,  Field  Artillery).1 2  Starting  in  January  2002,  the  Army  replaced  the  unit- 
weighted  composites  with  a  new  set  of  nine  AA  composites  based  on  empirically  estimated 
regression  weights  derived  from  criterion-related  validities  for  the  Armed  Services  Vocational 
Aptitude  Battery  (ASVAB)  (Greenston,  Rumsey,  Zeidner,  &  Johnson,  2001).  With  support 
from  the  Army  Research  Institute  (ARI),  Zeidner,  Johnson,  and  colleagues  developed  the 
composites  as  part  of  a  proposed  two-tiered  classification  system  intended  to  replace  the  existing 
AAs  (see  Zeidner,  Johnson,  Vladimirsky,  &  Weldon,  2000,  2001).  In  Zeidner,  Johnson,  and 
colleagues’  original  conceptualization  of  this  system,  the  first  tier  was  intended  for  classifying 
recruits  to  one  of  150  entry-level  job  families.  The  second  tier,  aimed  at  a  smaller  set  of  job 
families  (9  or  17),  was  meant  for  vocational  counseling,  recruiting,  and  administrative  purposes. 
The  proposed  two-tiered  system  was  based  on  a  multi-year  program  of  simulation  research 
conducted  by  Zeidner,  Johnson,  and  others.  Results  of  their  research  indicated  that  the  proposed 
two-tiered  classification  system  (and  related  composites)  could  produce  substantial  gains  in 
aggregate  Soldier  performance  over  and  above  that  expected  using  the  existing  AAs  (Johnson, 
Zeidner,  &  Leaman,  1992;  Statman,  1993;  Zeidner  et  al.,  2000, 2001). 

Purpose  of  Report 

The  current  report  is  the  second  of  two  reports  documenting  research  assessing  major 
components  of  Zeidner  and  colleagues’  proposed  two-tiered  A  A  system.  The  first  report  (Diaz, 


1  Unit  weights  were  based  on  rational  linkages  to  job  content  made  by  subject-matter-experts  (SMEs). 

2  In  descriptions  of  the  Zeidner,  Johnson,  and  colleagues’  method,  these  weights  are  frequently  referred  to  as  least 
squares  estimates  (LSE)  or  LSE  weights,  as  the  weights  are  empirically  estimated  using  conventional  ordinary  least- 
squares  (OLS)  regression. 
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Ingerick,  &  Lightfoot,  2004)  independently  replicated  the  Zeidner,  Johnson,  and  colleagues’ 
method  of  empirically  deriving  AA  composites,  including  the  nine  AA  composites  currently  in 
operational  use.  The  primary  purpose  of  this  second  report  is  to  summarize  research  evaluating 
the  efficacy  of  the  proposed  AA  system  to  meet  the  Army’s  classification  objectives.  More 
specifically,  the  goals  of  this  research  were  to: 

1 .  Evaluate  the  stability  and  classification  potential  of  the  proposed  AA  composites, 
particularly  the  17  and  150  relative  to  the  9  AAs. 

2.  Evaluate  the  proposed  job  family  structures  comprising  the  two-tiered  system  (9,  17, 
and  150),  including  the  potential  identification  of  alternative  job  family  structures 
displaying  greater  classification  efficiency. 

3.  Review  remaining  issues  related  to  implementing  the  proposed  system. 

In  summary,  the  current  research  aimed  to  appraise  both  the  process  and  products  (i.e.,  Pearlman, 
1980)  of  the  proposed  A  A  system  and  its  potential  for  operationally  achieving  Army 
classification  objectives.  In  keeping  with  this  goal  and  the  recommendations  of  other 
classification  researchers  (Pearlman,  1980;  Sackett,  1988),  we  placed  special  emphasis  on 
external  and  practical  criteria,  specifically  classification  efficiency,  when  evaluating  the  system. 

This  report  is  organized  as  follows.  First,  we  review  the  underlying  conceptual  basis  and 
describe  the  major  components  of  Zeidner,  Johnson,  and  colleagues’  proposed  system,  and  its 
current  operational  version.  Second,  we  summarize  and  discuss  our  results  from  a  preliminary 
investigation  of:  (a)  the  stability  of  the  proposed  regression-weighted  AA  composites;  and  (b)  the 
differential  validity  of  the  proposed  job  family  structures  (9,  17,  and  150).  Both  composite-test 
stability  and  the  differential  validity  of  proposed  job  families  have  implications  for  the 
classification  potential  of  the  proposed  two-tiered  system.  Third,  we  summarize  our  findings 
from  a  comprehensive,  empirical  analysis  designed  to  integrate  these  two  issues.  Specifically, 
this  joint  analysis  assesses  the  effects  of  composite  stability  and  differential  validity  on  the 
classification  efficiency  of  the  proposed  AA  system.3  Fourth,  and  finally,  we  conclude  the  report 
with  a  brief  review  of  major  findings,  a  discussion  of  remaining  implementation  issues,  and 
suggestions  for  future  research. 


3  Consistent  with  Zeidner,  Johnson,  and  colleagues,  classification  efficiency  is  measured  using  standardized  mean 
predicted  performance  (MPP).  MPP  is  an  index  of  the  average  predicted  job  performance  of  n  recruits  expressed  in 
standard  deviation  units  (i.e.,  an  average  z-score). 


OVERVIEW  OF  ZEIDNER,  JOHNSON,  AND  COLLEAGUES’  PROPOSED  TWO- 

TIERED  CLASSIFICATION  SYSTEM 


The  underlying  conceptual  basis  for  Zeidner,  Johnson,  and  colleagues’  proposed 
classification  system  comes  from  Differential  Assignment  Theory  (DAT)  (Zeidner  &  Johnson, 
1994;  Zeidner,  Johnson,  &  Scholarios,  1997).  Originating  with  research  by  Horst  (1954,  1955) 
and  Brogden  (1959),  DAT  makes  two  fundamental  propositions.  First,  consistent  with  specific 
aptitude  theories  of  intelligence,  DAT  postulates  that  people  and  jobs  can  be  differentiated  on  the 
basis  of  specific  aptitudes  and  abilities.  Second,  DAT  emphasizes  that  within  a  multiple  job 
context,  such  as  Army  classification  and  assignment,  increasing  differential  validity,  as  opposed 
to  incremental  validity,  maximizes  classification  efficiency.  Factors  that  influence  differential 
validity,  and  indices  of  differential  validity,  include:  (a)  the  number  of  jobs  (m)  to  which 
individuals  can  be  assigned:  (b)  the  average  predictive  (or  criterion-related)  validity  of  those  jobs 
(R);  and  (c)  intercorrelations  among  test  composites  predicting  performance  in  those  jobs  (r) 
(Brogden,  1959;  Zeidner  &  Johnson,  1994;  Zeidner  et  al.,  1997).  Taken  together,  DAT  posits 
that  when  assigning  individuals  to  multiple  jobs,  performance  for  the  group  (as  a  whole)  will  be 
optimized  by  using  a  multidimensional  test  battery  and  a  set  of  differentially-weighted 
composites  tailored  to  specific  job(s).  The  practice  of  matching  individual  skills  and  task 
requirements  also  has  a  long,  informal  history.  The  designation  of  the  best  throwers  as 
“pitchers”  in  Little  League,  and  the  best  catchers  as  “basemen”,  is  one  common  example  of  this 
practice. 

Using  DAT  as  a  basis,  and  building  on  a  multi-year  program  of  research,  Zeidner  and 
colleagues  proposed  a  two-tiered  classification  system  (see  Greenston  et  al.,  2001;  Zeidner  et  al., 
2000, 2001).  The  two-tiered  system  consists  of  two  major  components:  (a)  differentially-weighted 
test  composites  tailored  to  specific  jobs  (or  job  families);  and  (b)  groupings  of  jobs  with  comparable 
aptitude  and  performance  requirements  into  families  or  Aptitude  Areas  (AAs).  Each  tier 
encompasses  both  components.  As  originally  conceived,  the  first  tier  was  intended  for  classifying 
recruits  to  one  of  150  entry-level  job  families.  The  second  tier,  aimed  at  a  smaller  set  of  job  families 
(9  or  17),  was  meant  for  vocational  counseling,  recruiting,  and  administrative  purposes. 

Based  on  their  intended  purpose,  classification  or  counseling,  test  composites  are  estimated 
differently.  When  estimating  composites,  weights  are  derived  from  predictive  (or  criterion-related) 
validities  corrected  for  range  restriction  and  criterion  unreliability.  Depending  on  the  tier  and  the 
intended  use  of  scores  based  on  the  composites,  the  appropriate  reference  population  (i.e..  Youth  or 
Army  Input)  for  making  these  corrections  differs.4  For  instance,  corrections  to  the  Youth  population 
are  appropriate  when  scores  are  used  to  determine  the  mental  eligibility  of  high  school  seniors  for 
Army  service.  Conversely,  corrections  to  the  Army  Input  population  are  most  appropriate  when 
using  scores  to  assign  recruits  to  entry-level  jobs.  In  sum,  the  specific  estimation  procedure 
employed  when  deriving  the  weights  depends  on  the  types  of  decisions  for  which  scores  based  on  the 
composites  are  intended  to  support. 


4  The  Youth  population  represents  all  18  -  23  year  olds  in  the  U.S.  population.  The  Army  Input  population 
represents  all  Army  recruits  who  pass  basic  service  qualifications,  and  are  therefore,  eligible  for  assignment  to  entry- 
level  MOS. 
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The  “operational”  version  of  the  proposed  two-tiered  system  functions  somewhat 
differently  than  the  version  originally  proposed  by  Zeidner  and  colleagues.  The  primary 
difference  is  that  under  this  system,  both  tiers,  instead  of  just  the  first  tier,  are  used  in  classifying 
recruits  to  entry-level  MOS.  Specifically,  the  second  tier,  consisting  of  either  9  (or  17)  AAs  and 
corresponding  test  composites,  is  used  to  determine  a  recruit’s  eligibility  for  assignment  to  an 
MOS.  Consequently,  these  composites,  and  their  respective  weights,  are  referred  to  as 
assignment  (AA)  composites  (or  AA  weights).  The  first  tier,  comprised  of  150  job  families  and 
corresponding  composites,  is  used  for  estimating  a  recruit’s  predicted  performance.  First  tier 
composites,  and  their  respective  weights,  are  referred  to  as  predicted  performance  (PP ) 
composites  (or  PP  weights).  Predicted  performance  scores  are  used  operationally,  in  conjunction 
with  AA  scores,  to  inform  decisions  as  to  where  to  assign  recruits,  so  as  to  optimize  overall 
aggregate  Soldier  performance. 

Because  the  primary  motivation  for  the  current  research  is  on  the  AA  composites  and 
corresponding  job  family  configurations,  our  analyses  focused  on  the  “operational”  version  of 
the  Zeidner  and  colleagues’  two-tiered  classification  system  and  its  efficacy  in  achieving  Army 
classification  objectives. 


5  Those  job  families  making-up  the  first  tier  are  basically  individual-level  MOS. 
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PRELIMINARY  INVESTIGATION  OF  APTITUDE  AREA  (AA)  COMPOSITES  AND 
JOB  FAMILY  STRUCTURE:  ISSUES  OF  STABILITY  AND  DIFFERENTIAL  VALIDITY 

Two  long-standing,  albeit  related,  issues  regarding  the  classification  potential  of 
regression-weighted  test  composites  are  their  stability  and  differential  validity.  By  stability,  we 
are  referring  to  the  degree  to  which  estimates,  in  this  case  OLS-derived  regression  weights, 
accurately  reflect  an  underlying  population  value  (i.e.,  are  generally  free  of  sampling  error).  By 
differential  validity,  we  are  referring  to  between-job  family  differences  in:  (a)  the  predictive  (or 
criterion-related)  validities  (R)  of  a  test  composite;  and  (b)  the  degree  to  which  test  composites, 
and  their  predicted  performance  scores,  are  intercorrelated  (r).6  In  a  classification  context, 
differential  validity  is  maximized  when  predictive  validities  differ  by  family  and  the 
intercorrelations  among  test  composites  are  low.  More  importantly,  when  making  operational 
decisions  about  the  classification  potential  of  one  or  more  test  composites,  these  differences 
should  be  attributable  to  systematic  between-family  differences  in  job  content  and  performance 
requirements  and  not  other  extenuating  factors  such  as  estimation  (i.e.,  sampling)  or 
measurement  error. 

While  the  implications  of  sampling  error  on  personnel  selection  decisions  are  well-known 
in  applied  psychology  (Hunter  &  Schmidt,  1990;  Schmidt  &  Hunter,  1977),  there  is  comparatively 
less  attention  to  this  issue  as  it  pertains  to  classification.  In  the  context  of  classification,  sampling 
error  can  lead  to  inaccurate  composite  estimates,  which  in  turn  may  artificially  inflate  (or  deflate) 
estimates  of  differential  validity.  Practically,  this  means  that  jobs  may  appear  to  be  more  or  less 
different  than  they  actually  are.  When  making  operational  decisions,  this  instability  is  likely  to 
bias  estimates  of  a  composite’s  classification  potential,  as  measures  of  classification  effectiveness, 
such  as  mean  predicted  performance  (MPP)  are  typically  based  on  one  or  more  indices  of 
differential  validity  (see  Brogden,  1959;  Zeidner  et  al.,  1997).  That  is,  the  higher  the  differential 
validity  across  a  set  of  jobs,  the  greater  will  be  the  estimated  classification  potential  of  a  set  of 
composites  specifically  tailored  to  those  jobs.  However,  if  estimates  of  differential  validity  are 
biased  (positively  or  negatively)  leading  to  inaccurate  estimates  of  classification  potential,  then 
decisions  regarding  the  operational  utility  of  a  set  of  composites  will  be  adversely  affected. 
Presently,  there  is  a  long-standing  debate  about  the  degree  to  which  regression-weighted 
composites  tailored  to  specific  jobs  based  on  test  batteries  assessing  specialized  aptitudes  and 
abilities,  such  as  the  ASVAB,  produce  differences  in  validities  that  represent  “true”  between-job 
differences  and  not  differences  due  to  sampling  error  or  other  artifacts  (Hunter,  1983, 1985; 
Hunter,  Crosson,  &  Friedman,  1985;  Schmidt,  Hunter,  &  Larson,  1988;  Schmidt,  Hunter,  & 
Pearlman,  1981;  Zeidner  &  Johnson,  1994;  Zeidner  et  al.,  1997). 

As  an  initial  step  in  evaluating  the  proposed  AA  system,  we  first  investigated  these 
issues.  In  addition  to  rationally  examining  the  methods  for  deriving  the  composites  and  job 
family  structures,  we  conducted  several  exploratory  analyses  designed  to  answer  the  following 
questions: 

1 .  To  what  extent  are  both  the  individual  regression  weights  making  up  the  composites, 
and  the  composites  (as  a  whole),  stable  and  largely  free  of  sampling  error? 


6  Differential  validity  is  synonymous  with  differential  prediction. 
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2.  Since  test  composites  are  tailored  to  specific  job  families,  to  what  degree  do 
composites  differ  across  the  proposed  job  family  configurations?  That  is,  is  there 
differential  validity?  If  not,  are  there  alternative  job  family  configurations  that  might 
increase  differential  validity  and,  thereby,  classification  efficiency? 

We  will  summarize  our  findings  for  each  in  turn. 

Stability  of  Proposed  Aptitude  Area  (AA)  Composites 

In  an  earlier  report  (see  Diaz  et  al.,  2004),  we  successfully  replicated  Zeidner,  Johnson, 
and  colleagues’  method  for  estimating  the  proposed  OLS-weighted  composites.  In  this  section, 
we  evaluate  the  stability  of  those  weights.  The  section  is  organized  as  follows.  First,  we  briefly 
review  Zeidner  and  colleagues’  method  for  deriving  the  proposed  AA  composites,  noting  its 
implications  for  composite  stability  and  more  practically,  classification  efficiency.  Second,  we 
summarize  our  findings  from  a  series  of  exploratory  analyses  aimed  at  empirically  assessing  the 
stability  of  the  proposed  test  composites. 

Overview  of  Zeidner  and  Colleagues’  Method  for  Deriving  Regression-Weighted  AA  Composites 

In  general,  Zeidner  and  colleagues’  method  involves  estimating  weights  from  corrected 
ASVAB  intercorrelations  and  criterion-related  validities  using  standard  OLS  regression.  All 
weights  are  job-  or  AA-specific,  meaning  that  weights  are  estimated  separately  for  each  MOS  or 
a  family  of  MOS  with  comparable  job  requirements  to  produce  composites  targeted  to  that 

specific  MOS  (or  job  family).  In  accordance  with  Differential  Assignment  Theory  (DAT), 

empirically-estimated  weights  are  expected  to  differ  across  jobs  (i.e.,  exhibit  differential  validity) 
in  meaningful  ways  that  capture  systematic  job-to-job  differences  in  content  and  performance 
requirements,  just  as  the  unit-weighted  composites  did  with  dichotomously  assigned  weights 
(0,1)  based  on  a  rational  analysis  of  job  requirements.  As  a  result,  MOS-specific  composites 
should  differentially  predict  Soldier  performance  for  that  MOS  (or  job  family). 

Using  this  approach,  Zeidner  and  colleagues  derived  two  sets  of  regression-weighted  test 
composites  for  use  under  the  AA  system.  The  first  set  of  test  composites,  and  their  respective 
weights,  is  intended  for  computing  AA  scores  to  be  used  in  determining  recruit  eligibility  for 
assignment  to  an  MOS.  These  composites  constitute  the  second  tier  of  Zeidner  and  colleagues 
proposed  two-tiered  classification  system.  The  second  set  of  test  composites,  and  their 
respective  weights,  is  intended  for  computing  predicted  performance  scores.  Operationally, 
predicted  performance  scores  inform  assignment  decisions,  in  conjunction  with  the  AA  scores. 
These  weights  form  the  first  tier  of  the  proposed  two-tiered  classification  system.  In  addition  to 
informing  Army  classification  decisions,  these  scores  have  frequently  been  used  by  Zeidner  and 
colleagues  for  basic  research  purposes  when  evaluating  classification  system  design  issues. 

As  discussed  earlier,  regression-weighted  composites  are  potentially  problematic  for 
classification  purposes  because  regression-based  estimates  are  sensitive  to  sampling  error 


7  Correlations  are  corrected  for  criterion  unreliability  and  range  restriction,  as  observed  correlations  are  based  on 
predictor-criterion  data  truncated  by  selection  and  classification  effects.  _ 
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(Cohen,  Cohen,  Aiken,  &  West,  2003;  Pedhauzer,  1997).  This  means  that  regression-based 
estimates  are  likely  to  capitalize  on  sample-specific  variance  (Hunter  et  al.,  1985;  Schmidt  et  al., 
1988).  To  address  this  issue,  Zeidner  and  colleagues  aggregate  jobs  (MOS)  to  ensure  sample 
sizes  (ns)  of  200  or  greater  when  estimating  composite  weights.  Estimates  based  on  this  design 
are  expected  to  be  generally  free  of  error  (see  Zeidner  et  al.,  2000,  2001)  and,  therefore, 
representative  of  the  “true”,  underlying  population  values. 

While  the  aforementioned  procedure  should  (and  likely  does)  decrease  estimation  error, 
there  are  several  reasons  for  investigating  this  issue  more  thoroughly.  First,  many  of  the 
sample  sizes  (ns)  on  which  weights  and  composites  are  based,  specifically  those  comprising 
the  first-tier,  still  fall  below  recommended  levels  (Maxwell,  2000).  While  increasing  n 
minimizes  the  standard  errors  associated  with  these  weights,  since  error  is  inversely  related  to 
n,  such  increases  do  not  completely  eliminate  error.  This  is  problematic  because  even 
relatively  small  standard  errors  can  distort  conclusions  about  differential  validity.  For 
example,  consider  two  weights  for  the  same  predictor  for  two  different  jobs,  .08  (Job  A)  versus 
.11  (Job  B).  Assuming  no  error,  the  weight  for  Job  B  is  37.5%  larger  than  that  of  Job  A, 
suggesting  that  the  attribute  underlying  the  predictor  is  more  important  to  the  performance  of 
Job  B  than  Job  A.  Given  these  data,  there  is  evidence  for  differential  validity.  Now,  assume 
that  the  standard  error  associated  with  both  weights  is  small,  about  .02.  This  means  that  the 
“true”  weight  for  Job  A  is  somewhere  between  .06  and  .10,  while  the  “true”  weight  for  Job  B  is 
between  .09  and  .13.  Given  this  new  information,  the  weights  now  suggest  two  alternative,  but 
differing,  conclusions.  The  first  is  that,  since  there  is  overlap  in  the  estimates,  differential 
validity  is  close  to  zero  because  the  observed  difference  in  the  weights  is  artifactual  (i.e.,  due 
to  error).  The  second  conclusion  is  that  the  “true”  weights  may  actually  differ  more  than  the 
observed  difference  suggests,  indicating  that  differential  validity  is  greater,  and  potentially 
more  than  double,  that  in  the  original  example.8 

This  situation  becomes  especially  problematic  when  observed  weights  are  close  to  zero, 
since  once  error  is  taken  into  account  this  suggests  that  the  “true”  population  weights  are 
basically  zero;  there  is  no  relationship  between  the  predictor  in  question  and  job  performance. 
Therefore,  any  observed  differences  between  these  weights  and  those  that  are  technically  non¬ 
zero,  but  otherwise  represent  small  effects,  are  likely  to  be  misleading.  As  evident  from  this 
illustration,  even  relatively  small  standard  errors  can  adversely  impact  estimates  and  conclusions 
about  differential  validity,  and  more  practically,  classification  efficiency.  A  cursory  review  of 
the  proposed  weights  and  composites,  particularly  for  the  first-tier  based  on  a  150  job  family 
configuration,  shows  that  there  are  many  cases:  (a)  where  the  observed  differences  in  the  weights 
for  the  same  ASVAB  subtest  across  families  is  sufficiently  small  to  potentially  capitalize  on 
error;  and  (b)  where  observed  weights  are  relatively  close  to  zero. 

The  second  reason  for  further  investigating  the  stability  of  the  test  composites  is  the  high 
level  of  collinearity  present  among  the  predictors,  the  ASVAB  subtests.  Collinearity,  or  in  this 
case  multicollinearity,  refers  to  the  degree  to  which  a  set  of  predictors  correlate  with  each  other. 


s  Note  the  same  issues  apply  when  considering  differential  validity  as  it  applies  to  intercorrelations  among  the 
composites  and  corresponding  predicted  performance  scores  (r).  Since  correlations  are  biased  due  to  sampling  error, 
the  greater  the  sampling  error,  the  higher  the  intercorrelation  among  predicted  performance  scores  (and  the  lower  the 
differential  validity  among  the  composites). 
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When  multicollinearity  is  high,  standard  errors  are  inflated  (Cohen  et  al.,  2003,  Pedhauzer, 

1997).  This  produces  a  condition  known  as  “bouncing  betas”,  whereby  the  magnitude,  and  even 
the  direction,  of  the  weights  changes  depending  on  which  predictors  are  included  in  the 
regression  model.  Intercorrelations  among  the  ASVAB  subtests  comprising  the  current  7-test 
battery  tend  to  be  uniformly  high,  ranging  from  .41  to  .83  in  the  Youth  population  (Mitchell  & 
Hanser,  1984).9  When  working  with  the  Army  Input  population,  intercorrelations  among  the 
ASVAB  subtests  at  the  MOS-  and  job  family-level  are  roughly  comparable  in  magnitude. 
Therefore,  the  level  of  multicollinearity  present  in  the  ASVAB  subtests  is  likely  to  contribute  to 
inflated  standard  errors  when  estimating  composite  weights.  While  increased  n  is  associated 
with  smaller  standard  errors,  even  large  n  is  not  expected  to  fully  negate  the  influence  of 
collinearity  on  regression-based  estimates. 

In  summary,  additional  research  on  test  composite  stability,  and  its  implications  for  the 
proposed  classification  system,  is  needed.  The  sample  sizes  (n)  employed  during  estimation  and 
the  multicollinearity  present  in  the  ASVAB  could  not  only  separately,  but  also  jointly,  contribute 
to  inflated  standard  errors.  The  magnitude  of  these  errors,  even  those  that  are  relatively  small,  is 
practically  important  as  subsequent  conclusions  regarding  the  differential  validity  of  the  test 
composites  could  be  impacted.  To  address  these  issues,  we  conducted  a  series  of  exploratory, 
descriptive  analyses  to  empirically  assess  the  stability  of  the  proposed  test  composites.  All 
analyses  were  based  on  estimates  originally  derived  by  Zeidner,  Johnson,  and  colleagues 
(Zeidner  et  al.,  2000,  2001)  and  successfully  replicated  by  Diaz  et  al.  (2004).  Estimates  were 
based  on  data  contained  in  the  Skills  Qualification  Test  (SQT)  program  database.  This  database 
contains  ASVAB  subtest  scores  and  standardized  Skilled  Qualification  Test  (SQT)  scores  for 
Army  enlisted  personnel  covering  FYs  1987-1989  (N  =  257,810).  These  data  were  originally 
provided  by  the  U.S.  Army  Research  Institute  (ARI). 

Results  of  Exploratory  Analyses  Describing  the  Stability  of  the  Proposed  Test  Composites 

To  assess  the  stability  of  the  test  composites  and  their  respective  weights,  we  conducted 
two  sets  of  descriptive  analyses.  The  first  looked  at  the  magnitude  of  the  effect  sizes  associated 
with  the  composites  and  their  respective  weights  to  identify  “weak”  composites,  that  is 
composites  whose  weights  were  generally  not  significantly  different  from  zero.  More 
specifically,  the  purpose  of  these  analyses  was  to  detect  test  composites  that,  by  virtue  of  having 
effect  sizes  not  significantly  different  from  zero,  were  generally  unstable  and  thereby  could  be 
expected  to  bias  estimates  of  differential  validity.  The  second  set  of  descriptive  analyses,  signal- 
to-noise,  assessed  the  effects  of  collinearity  and  data  truncation  on  the  composites.  Comparable 
to  the  first  set  of  analyses,  the  purpose  of  the  signal-to-noise  analyses  was  to  identify  problematic 
composites  that  could  substantially  bias  inferences  about  differential  validity.  We  will 
summarize  each  in  turn. 

Identifying  Weak  Composites.  For  these  analyses,  we  took  the  estimated  composites  and 
their  respective  weights,  specifically  the  first-tier  weights  for  the  150  job  families  and  the 
second-tier  weights  for  the  17  and  9  job  families,  and  calculated  an  applicable  significance  test. 


9  These  intercorrelations  are  used  when  making  range  restriction  corrections  to  ASVAB-criterion  validities  to 
estimate  the  first-tier  weights  and  composites. 
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For  the  individual  weights,  we  computed  conventional  t-tests,  estimating  the  degree  to  which  the 
observed  values  were  significantly  different  from  zero.  For  the  full  composites,  we  conducted 
chi-square  tests  of  significance  estimating  the  degree  to  which  the  test  composite  (as  a  whole) 
differed  from  zero.10  When  conducting  the  tests,  we  considered  the  standard  error  of  the  weight 
or  the  composite,  respectively.  That  is,  all  tests  estimated  the  degree  to  which  observed  values 
were  significantly  different  from  zero,  taking  into  account  the  applicable  standard  error.  This 
enabled  us  to  distinguish  between  those  weights  and  composites  whose  difference  from  zero  was 
most  likely  artifactual  (due  to  error)  versus  those  whose  effect  size  was  technically  small,  but 
otherwise  stable.  From  these  analyses,  we  observed  the  following. 

First,  as  expected,  there  was  a  strong  linear  relationship  between  sample  size  («)  and 
composite  stability.  Tables  1  and  2  summarize  the  number  (and  percentage)  of  job  families 
exhibiting  non-significant  weights  by  job  family  configuration  and  ASVAB  subtest. 1 1  As 
evident  from  Tables  1  and  2,  as  the  number  of  job  families  increased,  and  thereby  n  decreased, 
the  number  of  non-significant  weights,  controlling  for  error,  likewise  increased.  While  the 
effects  of  small  n  were  minor  at  the  9  and  17  job  family  configurations,  they  were  most 
pronounced  for  the  1 50  job  family  configuration  (see  Table  2),  where  for  40.7%  of  the  job 
families  more  than  half  of  the  weights  making-up  their  respective  composite  were  not 
significantly  different  from  zero  (p  <  .05).  As  confirmation  of  this,  the  relationship  between  n 
and  the  number  of  nonsignificant  weights  among  the  150  job  families  was  strongly  negative  (r=- 
.673 ,p  <  .0001,  N  =  150),  indicating  that  as  n  decreases,  the  number  of  non-significant  weights 
in  a  test  composite  increases.  Therefore,  while  weights  for  the  9  and  17  job  family 
configurations  are  reasonably  robust  (i.e.,  significantly  different  from  zero),  this  was  not  the  case 
for  a  sizeable  percentage  of  the  150  job  families. 


Table  1 

Number  (and  Percentage)  of  Job  Families  with  Non-Significant  Weights  and  Composites  by 


GS 

AR 

AS 

MK 

MC 

El 

VE 

Full 

Composite 

JF 

N(%) 

N(%) 

N(%) 

N(%) 

N(%) 

N(%) 

N(%) 

N(%) 

9 

0(0.0) 

0(0.0) 

0(  0.0) 

0(0.0) 

0(0.0) 

0(0.0) 

0(0.0) 

0(0.0) 

17 

5(29.4) 

0(0.0) 

0(0.0) 

0(0.0) 

0(0.0) 

1(5.9) 

0(0.0) 

0(0.0) 

150 

119(79.3) 

33(22.0) 

46(30.7) 

31(20.7) 

74(49.3) 

76(50.7) 

67(44.7) 

0(0.0) 

1  YL/tC.  X  law  y  UllU  X  /  jvv  v  — - - - - o - - ______  J. 

constraint.  Significance  set  at  p<  .05  (two-tailed).  GS  =  General  Science;  AR  =  Arithmetic  Reasoning; 
AS  =  Auto  &  Shop  Information;  MK  =  Mathematical  Knowledge;  MC  =  Mechanical  Comprehension;  El 
=  Electronics  Information;  VE  =  Verbal. 


10  Given  the  n  involved,  these  chi-square  tests  are  equivalent  to  the  standard  overall  F-test  in  regression. 

11  For  significance  tests  results  by  job  family  and  ASVAB  subtest,  see  Appendix  A. 
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Table  2 


I  itLFIV'  ^ 

Number  (and  Percentage)  of  Job  Families  with  Non-Significant  Weights  within  their 
Respective  Composite  by  Job  Family  Configuration 

xt _ i _ f\i„. 


Number  of  Non-Significant  Weights 


0 _ 1 _ 2 _ 3 _ 4 _ 5 _ 6 

JF _ N(%)  N(%)  N(%)  N(%)  N(%)  N(%)  N(%) 

9  9(100.0)  0(0.0)  0(0.0)  0(0.0)  0(0.0)  0(0.0)  0(0.0) 

17  12(70.6)  4(23.5)  1(5.9)  0(0.0)  0(0.0)  0(0.0)  0(0.0) 

150  7(4.7)  21(14.0)  29(19.3)  32(21.3)  40(26.7)  16(10.7)  5(3.3) 

Note.  For  the  9  and  17  job  family  configurations,  results  exclude  weights  fixed  to  zero  because  of  positive 
constraint.  Significance  set  at  p  <  .05  (two-tailed). 


The  second  trend  is  that  composite  stability  was  related,  over  and  above  the  effects  of  n, 
to  the  particular  ASVAB  subtest  and  job  family.  As  can  be  seen  from  Table  1,  the  GS  subtest 
was  consistently  associated  with  small  effect  sizes,  particularly  at  the  150  job  family 
configuration,  where  79.3%  of  the  job  families  displayed  a  non-significant  weight  for  GS. 
Inspection  of  the  actual  weights  confirmed  this,  as  GS  repeatedly  corresponded  to  lower  effect 
sizes  relative  to  the  other  subtests,  even  when  weights  were  technically  significant.  At  the  150 
job  family  configuration,  El  (50.7%  non-significant),  MC  (49.3%),  and  VE  (44.7%)  also 
emerged  as  ASVAB  subtests  that  tended  to  be  associated  with  non-significant  effect  sizes.  This 
suggests  that:  (a)  the  ability  of  some  subtests  to  differentiate  among  the  job  families  (as  currently 
constructed)  is  minor;  and  (b)  some  of  the  observed  differentiation  for  a  large  number  of  families 
at  the  150  level  is  largely  artifactual  (due  to  error).  As  for  individual  job  families,  a  visual 
inspection  of  plots,  showing  the  standard  errors  of  composite  weights  organized  by  n  and 
ASVAB  subtest  for  the  1 50  job  family  configuration,  indicated  that  the  magnitude  of  the  error 
varied  among  job  families  with  comparable  n.  That  is,  holding  n  and  ASVAB  subtest  constant, 
some  job  families  were  associated  with  larger  standard  errors  than  others.  This  indicates  that 
sampling  error  differentially  affected  composite  estimates  for  the  different  job  families,  as  would 
be  expected  given  that  the  effects  of  sampling  error  are  random. 

In  sum,  we  identified  a  number  of  composites,  mainly  at  the  150  job  family  configuration, 
that  are  weak.  That  is,  controlling  for  error,  there  were  a  number  of  composites  containing  weights 
that  were  not  significantly  different  from  zero.  Larger  standard  errors  tended  to  be  a  function  of: 
(a)  low  n;  (b)  the  ASVAB  subtest  corresponding  to  the  weight;  and/or  (c)  sampling  error.  These 
findings  are  practically  important  for  two  reasons.  First,  they  indicate  that  for  a  number  of  families 
at  the  150-level  there  are  few  ASVAB  subtests  to  meaningfully  differentiate  them  from  other  jobs. 
Second,  because  of  the  number  of  composites  that  are  weak,  estimates  of  differential  validity, 
particularly  for  those  weights  relatively  close  to  zero,  will  either  over-  or  under-estimate  “true’ 
differences  between  job  families. 

Signal  to  Noise.  We  also  conducted  diagnostics  to  examine  the  impact  of  collinearity 
among  ASVAB  tests  on  the  quality  of  the  estimated  weights  of  the  composites  using  the 
approached  described  by  Belsley  (1988).  This  approach  employs  a  signal-to-noise  measure  that 
jointly  accounts  for  two  sources  of  “weakness”  in  the  data,  namely,  collinearity  and  “short  data.” 
In  our  diagnostics,  the  “signal”  corresponds  to  the  unknown  population  values  of  the  composite 
LSE  weights,  while  noise  corresponds  to  error  in  estimating  these  weights.  Collinearity  and/or 
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short  data  can  lead  to  unstable  regression  weights  and  are  directly  relevant  in  our  problem.  First, 
the  high  correlations  among  AS  VAB  scores  raise  concerns  regarding  the  potentially  adverse 
effects  of  collinearity  on  the  composite  weights.12  Second,  restriction  in  range  due  to  sample 
selection  is  relevant  to  the  problem  of  short  data,  which  occurs  when  a  predictor  has  small  length 
(or  variance).13  ASVAB  tests  that  are  important  to  the  job  family  or  MOS  potentially  exhibit 
short  data  problem  since  they  are  impacted  most  by  selection-related  range  restriction,  with 
estimated  weights  with  the  “wrong”  sign. 

The  following  steps  summarize  the  “weak  data”  diagnostic  framework  of  Belsley  (1991), 
as  it  applies  to  the  current  problem.  These  diagnostic  steps  are  carried  out  separately  at  the 
MOS-  or  job  family-  level  data.  In  the  descriptions  below,  X  denotes  the  n  by  seven  matrix  of 
ASVAB  test  scores  of  individuals  belonging  to  the  MOS  or  job  family  under  consideration. 

1 .  Detecting  Collinearity  Problem.  The  collinearity  diagnostic  is  based  on  the  indices 

1A-7 

M'k 

where  juk  is  the  Mi  smallest  singular  value  of  X  (or  square-root  of  the  kth  smallest 
eigenvalue  of  X’X)  and  is  the  maximum  singular  value  of  X.  A  collinearity  problem 
exists  if  the  largest  index  7]}  is  greater  than  30.  Note  that  the  cut-off  value  30  is  the  same 
as  that  typically  used  when  identifying  important  dimensions  in  factor  analysis.  The 
number  of  linear  dependencies  (or  dimension  of  the  collinearity  problem)  is  equal  to  the 
number  of  values  greater  than  30.  Two  or  more  subtests  are  involved  in  a  linear 
dependency  if  the  variances  of  their  regression  weight  estimates  are  mostly  accounted  for 
by  the  same  linear  dependency. 

While  collinearity  can  be  expected  to  degrade  regression  weight  estimates  of  subtests  that 
are  involved  in  the  relevant  linear  dependencies,  by  itself  it  is  not  harmful.  If  there  is  a 
strong  enough  relationship  between  job  performance  criterion  and  ASVAB  subtests,  then 
the  regression  weight  estimates  would  still  be  reliable. 

2.  Signal-To-Noise  Diagnostic.  The  signal-to-noise  diagnostic  is  conceptually  based  on 

t2  =yffr[var(^)}~V 


12  A  distinction  is  made  in  this  approach  between  collinearity  and  correlation  or  statistical  relationship  in  general. 
Correlation  is  sufficient  but  not  necessary  for  collinearity.  A  collinearity  problem  is  characterized  by  near  “linear 
dependence”  among  two  or  more  predictor  variables.  Geometrically,  these  predictors  form  an  unstable  base  for  the 
regression  expectation  surface,  which  can  lead  to  unstable  weights. 

13  Correction  for  range  restriction  that  is  traditionally  applied  to  adjust  selection  sample  correlations  to  a  reference 
population  values  does  not  address  the  short  data  problem.  While  these  two  issues  are  not  exactly  equivalent,  they 
are  related.  Under  the  same  condition,  when  the  range  of  a  subtest  is  restricted,  the  small  variance  in  the  subtest 
leads  to  “difficulty”  in  estimating  the  subtest  weight. 


11 


where  j8  is  the  true  but  unknown  regression  weight  and  p  is  the  estimate  of  the 
regression  weight  with  variance  var(p).14  Computationally,  the  diagnostic  is  carried  out 

using  a  test  statistic,  based  entirely  on  the  LSE  estimate  fa ,  that  is  distributed  as  a  non¬ 
central  F  (see  equation  (7.12)  on  page  212  of  Belsley).  Conceptually,  large  values  of  the 
ratio  r2  (and  the  test  statistic)  indicate  that  there  is  a  sufficient  signal-to-noise  ratio.  This 
arises  when  the  unknown  regression  weight  /?  is  large  enough,  even  if  the  variance 

var(^)  is  inflated  due  to  collinearity  problems;  large  enough  values  also  could  arise  even 
if  P  is  small  (e.g.,  weak  criterion-subtest  relationship)  if  the  corresponding  predictor 

variable  is  not  involved  in  a  collinearity  problem  with  small  var^j . 

The  combination  of  the  collinearity  diagnostic  index  and  the  signal-to-noise  ratio 
triangulates  the  weak  data  problem  in  the  following  way.  First,  there  is  no  weak  data 
problem  if  the  signal-to-noise  ratio  is  significantly  high,  even  in  the  presence  of  linear 
dependencies  (i.e.,  T]k  values  above  30).  Collinearities  in  this  situation  are  considered 
not  harmful.  Second,  if  the  signal-to-noise  ratio  is  not  significant,  then  this  indicates  one 
of  two  weak  data  problems:  (1)  harmful  collinearity,  in  the  presence  of  linear 
dependencies;  (2)  or  short  data,  in  the  absence  of  linear  dependencies.  This  triangulation 
is  summarized  in  the  table  below: 


Inadequacy  of 
signal-to-noise 
diagnostic 


Presence  of  collinearity  as  indicated  by 
collinearity  diagnostic _ 


NO 

YES 

NO 

No  weak  data  problem 

Not  harmful  collinearity 

YES 

Short  data  problem 

Harmful  collinearity 
problem 

In  summary,  data  weakness  in  the  job  performance  equation  estimation  potentially  arises 
from  “correlated”  predictor  AS VAB  battery  and  “data  shortness”  (or  range  restriction)  due  to 
sample  selection.  If  weak  data  is  detected  then  there  is  not  enough  information  (or  power)  in  the 
sample  to  conclude  one  way  or  another  regarding  significance  or  the  nature  of  the  relationship 
between  a  subtest  and  job  performance  and  a  larger  sample  may  be  required. 


Results  of  Weak  Data  Diagnostics.  Table  3  summarizes  the  results  of  the  weak  data 
diagnostic  analysis  in  terms  of  number  of  ASVAB  subtests  involved  in  harmful  collinearity  and 
short  data  problem  by  job  family  configuration.15  Overall  the  result  of  this  analysis  is 
comparable  to  the  stability  analyses  above  based  on  the  usual  statistical  significance  test  on  each 
weight.  The  signal-to-noise  test  is  similar  in  nature  to  analysis  of  power  and  does  not  exhibit  the 
tendency  of  the  ordinary  significance  test  to  reject  the  null  hypothesis  of  zero  weight  when  the 
sample  size  is  sufficiently  large. 


14  Note  that  this  is  not  the  same  as  the  usual  test  of  significance  as  the  unknown  parameter  value  ft  instead  of 
appears  in  the  numerator. 
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The  weak  data  diagnostics  indicate  that  the  overall  quality  of  the  9  job  family  weights  is 
good.  Only  GS  exhibited  weights  that  were  adversely  affected  by  collinearity  or  short  data.  Our 
analysis  showed  that  GS  was  consistently  involved  in  a  linear  dependency  with  VE  in  five  out  of 
nine  job  families,  and  AR  with  MK  in  eight  out  of  nine  job  families.  However,  the  sample  sizes 
at  the  9  job  family  configuration  and  ASVAB-criterion  relationship  were  large  enough  to 
overcome  harmful  effects  of  collinearity  and  produce  composite  weights  that  could  be  reliably 
differentiated  from  noise. 


Table  3 

Number  (and  Percentage)  of  Job  Families  with  Weights  Involved  in  Harmful  Collinearity 
(HQ  andShortData  (SD)  Problem  by  ASVAB  Subtest  andJob  Family  Configuration 


\ - s  --  : 

Type  of 

No.  JF  Weak  Data 

GS  AR 

AS 

MK 

MC 

El  VE  Total 

N  (%)  N 

mm 

N 

mm 

N 

(%) 

N  (%) 

N  (%)  N  (%)  N  (%) 

9 

HC 

4(44.4)  0 

(0) 

0 

(0) 

0 

(0) 

0  (0) 

0  (0)  0  (0)  4  (6.3) 

SD 

3(33.3)  0 

(0) 

0 

(0) 

0 

(0) 

0  (0) 

1(11.1)  0  (0)  4  (6.3) 

Total 

7(77.8)  0 

(0) 

0 

(0) 

0 

(0) 

0  (0) 

1(11.1)  0  (0)  8(12.7) 

17 

HC 

7(41.2)  0 

(0) 

0 

(0) 

0 

(0) 

0  (0) 

0  (0)  2  (11.8)  9  (7.6) 

SD 

7(41.2)  0 

(0) 

2  (11.8) 

0 

(0) 

2(11.8) 

5(29.4)  0  (0)  16(13.4) 

Total 

14(82.4)  0 

(0) 

2  (11.8) 

0 

(0 L 

2(11.8) 

5(29.4)  2  (11.8)  25  (21) 

155 

HC 

93  (60)  52 

(33.5) 

4 

(2.6)  44  (28.4) 

1  (0.6) 

3  (1.9)  72  (46.5)269(24.8) 

SD 

59(38.1)  33 

(21.3)  88 

(56.8)  31 

(20)  1 15  (74.2)  126  (81.3)  27  (17.4)  479  (44.1) 

Total 

152(98.1)  85 

(54.8)  92 

(59.4)  75 

(48.4)  1 16  (74.8)  129  (83.2)  99  (63.9)  748  (68.9) 

Note.  GS  =  General  Science;  AR  =  Arithmetic  Reasoning;  AS  =  Auto  &  Shop  Information;  MK  = 


Mathematical  Knowledge;  MC  =  Mechanical  Comprehension;  El  =  Electronics  Information;  VE  =  Verbal. 


For  the  17  job  family  configuration,  the  diagnostics  again  suggest  good  quality  of  weights 
overall.  As  in  the  9  job  family,  the  GS  weights  were  the  most  adversely  affected  by  weak  data 
problems,  with  14  out  of  17  composite  weights  exhibiting  harmful  collinearity  or  short  data. 

Only  two  additional  composites,  CL1  and  ST2,  showed  weak  data  problems  on  two  ASVAB 
subtests,  AS  and  MC,  which  previously  were  not  identified  under  the  nine  job  family.  The  weak 
data  diagnostics  is  suggesting  that  the  subtest-criterion  relationship  for  AS  and  MC  simply  were 
not  strong  enough  in  CL1  and  ST2  jobs,  and  that  under  ASVAB  collinearity  conditions  these  two 
subtests  do  not  play  an  important  role  for  these  jobs.  Composite  weights  on  the  other  subtests 
could  be  reliably  differentiated  from  noise,  even  if  some  of  them  were  involved  in  linear 
dependencies  (e.g.,  AR  and  MK  in  14  out  of  17  job  families). 

Diagnostics  at  the  MOS  level,  which  should  closely  approximate  the  150  job  family 
configuration,  indicate  weak  data  problems  for  many  MOS-job  families  and  far  more  ASVAB 
subtests  than  observed  in  the  9  and  17  job  family  configurations.  GS  weights  continued  to 
exhibit  weak  data  problems  for  almost  all  MOS  at  this  level,  which  is  not  surprising  given  earlier 
observations.  However,  all  the  other  subtests  now  exhibit  weak  data  problems  for  at  least  48 
percent  of  the  MOS.  In  particular,  AR  and  MK  weights  are  now  susceptible  to  the  linear 
dependency  between  these  two  subtests,  while  before  there  was  large  enough  sample  size  to 
overcome  harmful  effects  of  collinearity  for  these  two  subtests.  The  same  can  be  observed  for 
the  other  ASVAB  collinearities.  Overall,  the  number  of  subtests  that  cannot  reliably  be 
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differentiated  from  noise  for  each  MOS-job  family,  combined  with  the  frequency  of  this 
occurrence  across  MOS-job  families,  indicate  that  true  differential  validity  cannot  be  achieved  at 
the  MOS  level;  that  is,  many  of  the  MOS-job  families  will  have  the  same  ASVAB  validity 
pattern  if  we  exclude  subtests  on  each  MOS-job  families  that  are  not  different  from  noise. 

In  summary,  the  weak  data  diagnostics  indicate  that  composite  weights  at  the  nine  and  17 
job  family  can  reliably  be  differentiated  from  noise  overall.  The  harmful  effects  of  ASVAB 
collinearity,  while  degrading  the  quality  of  some  of  the  weights,  were  not  strong  enough  to 
conclude  that  the  differential  validities  and  related  classification  properties  of  the  composite 
weights  are  of  no  value.  The  situation  is  very  different,  however,  at  the  155  MOS  level,  where 
the  high  frequency  of  MOS-job  family  composites  with  weak  composite  weights  on  several 
subtests  at  the  same  time  indicates  that  true  differential  validity  is  not  achievable  at  the  155  MOS 
or  150  job  family  configuration. 

Differential  Validity  of  Proposed  Job  Family  Structure 

As  described  earlier,  differential  validity  is  operationalized  as  between-job  differences  in 
ASVAB-predictive  (or  criterion-related)  validities  (. R ),  and  as  between-job  differences  in 
intercorrelations  among  test  composites  and  their  respective  predicted  performance  scores  (r).  For 
differential  validity  to  be  meaningful,  these  differences  need  to  reflect  systematic  differences  in  the 
underlying  performance  requirements  among  the  different  jobs.  Theoretically,  differential  validity 
is  expected  to  increase  when  jobs  (or  job  families)  are  substantially  different  from  each  other  in 
terms  of  their  actual  performance  requirements.  As  described  earlier,  composites  are  job  family- 
specific  in  that  each  composite  seeks  to  maximize  performance  prediction  for  that  particular  MOS 
or  family  of  MOS.  Therefore,  how  job  families  are  constructed  could  significantly  influence  die 
ability  of  the  composites  to  differentiate  between  recruits’  expected  performance  across  a  family  of 
jobs.  In  this  section,  we  evaluate  between-family  differences  in  the  proposed  composites,  and  their 
respective  weights.  The  section  is  organized  as  follows.  First,  we  briefly  review  Zeidner  and 
colleagues’  method  for  constructing  the  proposed  job  families,  noting  its  implications  for 
differential  validity,  and  more  practically,  classification  efficiency.  Second,  we  summarize  our 
findings  from  a  series  of  exploratory  analyses  designed  to  assess  the  degree  to  which  the  job 
families,  as  currently  constructed,  can  be  meaningfully  differentiated  using  the  proposed 
composites. 

Overview  of  Zeidner  and  Colleagues  ’  Method  for  Constructing  Job  Families 

There  are  a  number  of  different  methods  available  for  clustering  jobs.  Zeidner  and 
colleagues’  method  represents  a  hybrid  approach.  In  brief,  it  involves  clustering  MOS  into  families 
empirically  using  the  previously  derived  test  composites.  Using  conventional  cluster  analysis 
procedures,  MOS  are  allocated  to  families  in  a  way  that  minimizes  within- family  differences  in 
composites,  while  maximizing  between- family  differences.  For  practical  and  conceptual  reasons, 
some  modifications  are  then  made  to  the  placement  of  MOS  within  families  based  on  a  rational 
analysis  of  job  content.  This  method  has  produced  three  alternative  configurations  of  job  families. 
The  first  consists  of  150  job  families,  the  majority  of  which  are  individual  MOS.  This  configuration 
corresponds  to  the  first-tier  in  Zeidner  and  colleagues’  proposed  classification  system.  The  second 
and  third  configurations  consist  of  17  and  9  job  families  respectively,  each  family  consisting  of  a 
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group  of  comparable  MOS.  These  two  configurations  represent  alternative  versions  of  the  second- 
tier  of  the  proposed  classification  system. 

As  with  the  composites,  Zeidner  and  colleagues’  method  is  motivated  by  the  proposition 
that  jobs  differ  in  the  kinds  of  cognitive  aptitudes  and  abilities  required  for  successful  performance. 
Theoretically,  jobs  can  then  be  scaled  accordingly.  There  is  both  empirical  and  conceptual  support 
for  this  proposition  in  applied  psychology,  which  demonstrates  that  jobs  can  be  differentiated  by 
cognitive  ability,  including  specific  aptitudes  (DesMaris  &  Sackett,  1993;  Gottffedson,  1986). 
Equally  as  important,  there  is  evidence  that  differences  in  job  requirements  are  tied  to  differences 
in  validity.  Specifically,  jobs  requiring  higher  levels  of  an  aptitude  display  higher  levels  of 
predictive  (or  criterion-related)  validity  for  that  aptitude  (Hunter  &  Hunter,  1984).  Therefore,  there 
is  good  reason  to  expect  that  jobs  can  be:  (a)  meaningfully  differentiated  by  aptitude;  and  (b)  that 
said  differences  will  correspond  to  differences  in  predictive  validity.  Nevertheless,  additional 
research  on  the  proposed  job  family  configurations  is  needed.  There  are  several  reasons  for  this. 

The  first  reason  is  that  clustering  MOS  into  families  empirically  based  on  differences  in 
test  composites,  and  their  respective  weights,  potentially  capitalizes  on  error  associated  with  the 
composites.  As  described  and  documented  in  the  preceding  section,  there  is  instability  in  the 
composites,  and  their  respective  weights,  particularly  at  the  1 50  level.  Because  of  this  error,  jobs 
may  appear  more  or  less  similar  than  they  actually  are.  Using  composites,  then,  to  cluster  jobs 
may  lead  to  misleading  recommendations  about  where  to  best  place  MOS  within  job  families,  as 
a  means  to  maximize  differential  validity  and  ultimately  classification  efficiency. 

A  second  reason,  not  unrelated  to  the  first,  is  that  clustering  algorithms  are  prone  to 
capitalize  on  sample-specific  variance  (or  error).  That  is,  it  is  not  uncommon  for  job  cluster 
solutions  to  fail  to  cross-validate  when  using  a  different  sample  and/or  types  of  job-related  data 
than  that  from  which  the  original  solution  was  derived  (Pearlman,  1980;  Sackett,  1988;  Statman, 
Gribben,  Harris,  &  Hoffman,  1994).  In  the  first  place,  clustering  algorithms  depend  heavily  on 
internal  criteria,  specifically  some  mathematical  expression  of  observed  differences  relative  to 
other  differences  within  the  same  data.  Equally  problematic,  different  algorithms  use  different 
criteria.  This  explains  why  different  clustering  algorithms  tend  to  produce  widely  divergent  job 
structures,  even  when  using  the  same  data  (Lightfoot,  Diaz,  &  Vladimirsky,  1997;  Statman  et  al., 
1994).  Consequently,  results  can  and  do  vary  from  sample  to  sample  owing  to  the  idiosyncrasies 
within  a  particular  sample.  In  the  second  place,  clustering  techniques,  such  as  that  applied  by 
Zeidner  and  colleagues,  typically  lack  formal  statistical  significance  testing  frameworks  or  “rules 
of  thumb”  for  determining  the  reliability  and  practical  significance  of  observed  differences 
(Lightfoot  et  al.,  1997).  In  other  words,  the  procedures  are  primarily  exploratory.  Therefore, 
unlike  other  conventional  statistical  methods  (i.e.,  regression),  there  are  no  guides  for  minimizing 
Type  I  or  Type  II  errors  as  is  common  in  conventional  hypothesis  testing. 

In  summary,  while  there  is  some  basis  to  expect  that  the  proposed  job  families  will  be 
associated  with  between-family  differences  in  the  composites,  additional  research  is  needed. 
Specifically,  the  current  analyses  were  motivated  by:  (a)  the  sampling  error  present  in  the 
composites,  and  their  respective  weights;  and  (b)  the  potential  for  empirically-driven  clustering 
techniques  to  capitalize  on  this  error.  As  before,  the  magnitude  of  these  errors,  even  those  that 
are  relatively  small,  is  important.  Statistically,  error  could  influence  estimates  and  conclusions 
regarding  the  differential  validity  of  the  composites.  Practically,  these  conclusions  could  in  turn 
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impact  operational  decisions  about  which  job  family  configuration  is  optimal  for  classification 
purposes.  Therefore,  the  effectiveness  of  Army  classification  policy  is  directly  tied  to  the  quality 
of  the  evidence  used  to  form  the  conclusions  informing  these  decisions.  To  address  these  issues, 
we  conducted  a  series  of  exploratory,  descriptive  analyses  aimed  at  empirically  assessing  the 
differential  validity  of  the  proposed  job  families. 


Results  of  Exploratory  Analyses  Assessing  the  Differential  Validity  of  the  Proposed  Job  Families 

We  conducted  two  sets  of  descriptive  analyses  to  assess  the  degree  job  families  (as 
currently  constructed)  could  be  meaningfully  differentiated  by  the  proposed  test  composites. 

The  first  analyses  directly  targeted  between-family  differences,  controlling  for  sampling  error,  in 
the  composites  and  their  respective  weights  within  the  9  and  17  job  family  configurations.  That 
is,  the  first  set  of  analyses  focused  on  between-family  differences  in  predictive  (or  criterion- 
related)  validities.  The  second  set  of  analyses  investigated  between-family  differences  in 
predicted  performance  scores  based  on  the  test  composites  for  the  9  and  17  job  family 
configurations.  That  is,  the  second  set  of  analyses  focused  on  the  degree  to  which  test 
composites  were  intercorrelated,  such  that  they  produced  predicted  performance  scores  that  were 
generally  equivalent  across  job  families. 

Differences  in  Test  Composites  and  their  Individual  Weights.  To  directly  test  differences 
in  the  profile  of  weights  making  up  a  composite  and  different  weights  individually,  we 
conducted  a  nested  multiple  analysis  of  covariance  (MANCOVA)  followed-up  by  individual 
analyses  of  covariance  (ANCOVAs).  For  the  MANCOVA,  job  family  configuration  served  as 
the  independent  variables  and  weights  for  the  7  ASVAB  subtests  as  the  dependent  variables. 

To  take  into  account  estimation  error,  we  included  the  square  root  of  observed  n  (on  which 
estimates  were  based)  as  the  covariate.  To  ensure  that  weights,  when  aggregated  to  job  family- 
level,  reflected  values  observed  at  that  level,  estimates  were  weighted  by  acquisition  ns  -  the 
same  ns  used  to  aggregate  validities  to  the  job  family-level  when  originally  deriving  weights  for 
the  9  and  17  job  family  configurations.  Overall  results  for  the  MANCOVAs  were  meant  to  test 
between-family  differences,  holding  error  constant,  in  the  profile  of  weights  (i.e.,  the  full 
equation)  comprising  the  composite.  That  is,  how  much  of  the  variability  in  a  linear  profile  of 
the  weights  can  be  explained  by  between-family  differences?  To  test  between-family  differences 
in  specific  weights  associated  with  the  different  ASVAB  subtests,  we  followed  the  MANCOVA 
with  individual  ANCOVAs  based  on  the  same  model;  except  for  each  ANCOVA  there  was  now 
only  one  dependent  variable,  that  being  the  weights  for  the  applicable  ASVAB  subtest.  This 
two-stage  procedure  is  consistent  with  recommendations  for  conducting  multivariate  tests  of 
differences  in  multiple  dependent  variables  (Tabachnick  &  Fidell,  1996). 

Results  for  the  omnibus  tests  in  MANCOVA  showed  that,  controlling  for  sampling  error 
(i.e.,  4n  ),  there  were  significant  between-family  differences  in  the  profiles  of  the  weights  across 
job  families  for  both  the  9  (Wilk’s  A  (56, 710.768)  =  -389,  p  <  .001)  and  17  (Wilk’s  A  (S6: 710.768)  = 
.101,/?  <  .001)  job  family  configurations.  That  is,  for  both  the  9  and  17  job  family 
configurations,  between-family  differences  explained  a  significant  amount  of  the  variability  in 


15  The  model  is  nested  in  that  the  job  families  comprising  the  17  job  family  configuration  are  nested  within  the  9 
family  configuration. 
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the  profile  of  the  weights,  beyond  that  expected  by  within-family  differences  (i.e.,  random  error). 
At  the  level  of  the  ASVAB  subtests,  we  observed  the  following.  First,  that  particular  subtests 
were  more  strongly  associated  with  between-family  differences  than  others  (see  Table  4).  For 
example,  looking  across  the  two  configurations,  between-family  differences  accounted  for  the 
greatest  variability  (see  R2  values)  in  weights  for  Auto  &  Shop  Information  (AS),  Arithmetic 
Reasoning  (AR),  and  Verbal  (VE).  Second,  the  partial  effect  sizes  (R2 s)  for  the  17  job  family 


Table  4 

Resuite  of  Individual  AN  CO  VAs  by  ASVAB  Subtest  and  Job  Family  Configuration 


Subtest 

9 

Job  Family  Configuration 

17 

F 

df 

P 

R2 

F 

df 

P 

R2 

GS 

5.063 

8,137 

.001 

.228 

.950 

8,137 

.478 

AR 

13.076 

8,137 

.001 

.433 

3.224 

8,137 

.002 

.158 

AS 

23.875 

8,137 

.001 

.582 

4.122 

8,137 

.001 

.194 

MK 

7.159 

8,137 

.001 

.295 

2.530 

8,137 

.013 

.129 

MC 

7.202 

8,137 

.001 

.296 

1.763 

8,137 

.090 

.093 

El 

6.191 

8,137 

.001 

.266 

2.584 

8,137 

.012 

.131 

VE 

9.681 

8,137 

.001 

.361 

4.297 

8,137 

.001 

.201 

Note.  R2  values  are  partial  Rzs  and  reflect  the  unique  contributions  of  between-family  differences  associated 
with  a  particular  job  family  configuration  to  variability  in  the  applicable  composite  weights.  GS  =  General 
Science;  AR  =  Arithmetic  Reasoning;  AS  =  Auto  &  Shop  Information;  MK  =  Mathematical  Knowledge; 
MC  =  Mechanical  Comprehension;  El  =  Electronics  Information;  VE  =  Verbal. 

configuration  were  smaller  (roughly  50%+)  than  the  corresponding  values  for  the  9  job  family 
configuration,  indicating  most  of  the  differentiation  between  jobs  is  attributable  to  the  9  job 
families.  At  an  aggregate  level,  this  suggests  that  expanding  the  9  job  family  configuration  to  1 7 
families  does  further  differentiate  among  some  jobs,  but  it  is  unclear  if  that  added  differentiation 
is  practically  significant. 

In  summary,  controlling  for  sampling  error,  there  are  differences  in  composite  weights, 
both  as  a  set  and  individually,  across  job  families  in  the  9  and  17  job  family  configurations. 

There  were  two  additional  observations  of  note.  First,  consistent  with  earlier  findings  regarding 
composite  stability,  certain  ASVAB  subtests  were  more  strongly  associated  with  between-family 
differences  than  others  [i.e.,  Auto  &  Shop  Information  (AS)  versus  General  Science  (GS)]. 
Second,  expanding  the  9  job  family  configuration  to  17  appears  to  further  meaningfully 
differentiate  among  the  families,  but  the  degree  to  which  this  is  practically  significant  (i.e.,  leads 
to  substantial  increase  in  MPP)  was  unclear  from  the  present  findings. 

Differences  in  Predicted  Performance  by  Composite.  To  test  between-family  differences 
in  predicted  performance  scores  by  test  composite,  we  computed  distance  statistics  assessing 
differences  in  the  predicted  performance  scores  across  the  range  of  ability.  Mathematically,  the 
“distance”  between  two  composites,  indexed  by  i  and  j,  is  represented  by  the  following  formula: 

var(l'-^)=S,2+«;-2r(,.R,*r 
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As  the  left-hand-side  of  the  expression  indicates,  this  involved  estimating  predicted 
performances  based  on  the  two  composites  for  each  recruit,  and  taking  the  variance  of  their 
difference  in  the  recruit  population.  This  distance  statistic  is  a  useful  diagnostic  tool  for  assessing 
differential  validity-related  properties  of  the  composites.  Two  composites  that  are  more  or  less 
parallel  (i.e.,  no  differential  validity)  would  yield  an  intra-person  composite  difference  that  is 
fairly  constant  in  the  population,  as  would  be  indicated  by  a  variance  that  is  close  to  zero.  On  the 
other  hand,  if  the  composites  are  close  to  orthogonal,  then  the  variance  of  their  difference  would 
be  large.  In  our  analysis,  we  would  prefer  composites  that  are  dissimilar  from  each  in  the  sense 
that  variances  of  their  pairwise  differences  are  relatively  large.  The  aforementioned  ideas  also  are 
readily  verified  using  the  computational  formula  on  the  right-hand-side  of  the  expression  above. 
In  the  regression  context,  this  expression  is  proportional  to  the  loss  in  overall  R-square  when  two 
separate  regression  equations  from  two  samples  with  equal  sizes  are  combined.  This 
interpretation  is  not  appropriate  given  the  unequal  job  family  sample  sizes,  but  will  be  employed 
after  some  modifications  in  our  second  distance  statistic.  For  the  full  set  of  pairwise  distance 
values  for  the  9  and  17  job  family  configurations,  see  Appendix  C. 

For  the  9  job  family  configuration,  on  average,  the  Clerical  (CL)  (MD  =  4.147)  and 
Mechanical  Maintenance  (MM)  (MD  =  5.738)  job  families  displayed  the  largest  differences  with 
the  other  families  (see  Table  5).  Similarly,  the  largest  difference  between  any  two  families  was 
1 2.442  for  CL  versus  MM.  A  comparable  pattern  was  observed  for  the  1 7  job  family 
configuration.  The  two  families  exhibiting  the  largest  differences,  on  average,  were  Clerical  1 
(CL1)  (Md  =  6.313)  and  Mechanical  Maintenance  1  (MM1)  (MD  =  7.656).  The  largest  difference 
between  any  two  families  was  associated  with  CL1  versus  MM1  (20.242).  Overall,  differences 
were  larger,  on  average,  for  the  17  job  family  configuration  than  the  9  job  family  configuration. 


Table  5 


Differences  in  Predicted  Performance  Scores  by  Job  t  amuy 

C 

> 

Job  Family  Configuration 

>  17 

JF 

Mo 

Min 

Max 

JF 

Md 

Min 

Max 

CL 

4.147 

1.168 

12.442 

CL1 

6.313 

1.420 

20.242 

CO 

2.410 

.264 

6.423 

CL2 

2.392 

.529 

11.222 

EL 

1.346 

.332 

3.912 

COl 

4.046 

.848 

9.740 

FA 

1.630 

.264 

5.676 

C02 

1.629 

.194 

6.237 

GM 

1.827 

.325 

5.003 

ELI 

1.652 

.204 

6.669 

MM 

5.738 

1.946 

12.442 

EL2 

1.508 

.242 

5.387 

OF 

1.453 

.325 

4.010 

EL3 

2.225 

.218 

9.674 

SC 

1.592 

.267 

6.256 

FA 

1.906 

.195 

7.178 

ST 

1.789 

.267 

6.510 

GM1 

3.267 

.629 

9.673 

GM2 

2.095 

.194 

8.132 

MM1 

7.656 

2.751 

20.242 

MM2 

2.212 

.492 

7.222 

OF 

1.746 

.204 

7.231 

SC 

1.638 

.339 

8.023 

ST1 

1.876 

.195 

8.148 

ST2 

3.103 

.218 

12.012 

ST3 

1.765 

.284 

7.220 
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Note.  Distance  values  reflect  variance  of  differences  between  pairs  of  predicted  performance  scores 
multiplied  by  1 00. 

Table  6  (below)  provides  a  summary  comparison  of  differences  in  predicted  performance 
scores  across  the  9  and  17  job  family  configurations.  This  table  highlights  potential  increments  (or 
decrements)  in  differential  validity  by  shredding-out  some  of  the  9  job  families  into  17  families. 
When  assessing  the  gains  in  differential  validity  by  moving  from  the  9  to  a  17  job  family 
configuration  and/or  to  identify  possible  alternative  job  family  configurations,  we  focused  on  two 
criteria.  The  first  criterion  of  interest  was  whether  the  between-family  differences  for  the  same 
family  at  the  17  job  family  level  were  consistently  larger  than  the  corresponding  between-family 
differences  at  the  9  job  family  level.  As  evident  from  the  table,  only  General  Maintenance  (GM), 
Electronics  (EL),  and  Skilled  Technical  (ST)  appeared  to  show  consistent  gains  (on  average)  in 
differential  validity  from  shredding  out  their  respective  families  at  the  9  family  configuration  into 
2-3  smaller  families  for  the  17  configuration.  Results  for  the  other  families  were  mixed.  For 
example,  splitting  the  Clerical  (CL)  job  family  into  CL1  and  CL2  produced  a  sizeable  increment 
(on  average)  in  differential  validity  for  CL1  (from  MD  =  4.147  to  6.313),  but  a  relatively 
considerable  drop  for  CL2  (from  MD  =  4.147  to  2.392).  A  similar  pattern  is  evident  with  the 
Combat  (CO)  job  family. 


Table  6 

Com£arison  o[Distance  Statistics  Across  9  and  1 7  Job  Family  Configurations 


JF 

Md 

%+/- 

JF 

Md 

%+/- 

Clerical 

Electronics 

1.  CL  (9) 

4.147 

— 

1 .  EL  (9) 

1.346 

~ 

2.  CL1  vs.  CL2  (17) 

1.736 

-58.14 

2.  ELI  vs.  EL2  (17) 

.242 

-82.02 

3.CL1  vs.  Other  JFs  (17) 

6.618 

+59.59 

3.  ELI  vs.  EL3  (17) 

1.459 

+  8.40 

4.  CL2  vs.  Other  JFs  (17) 

2.436 

-41.26 

4.  EL2  vs.  EL3  (17) 

1.097 

-18.50 

Combat 

5.  ELI  vs.  Other  JFs  (17) 

1.767 

+31.28 

1.  CO  (9) 

2.410 

— 

6.EL2vs.  Other  JFs  (17) 

1.627 

+20.88 

2.  COl  vs.  C02  (17) 

1.543 

-35.98 

7.  EL3  vs.  Other  JFs  (17) 

2.360 

+75.33 

3.C01  vs.  Other  JFs  (17) 

4.213 

+74.81 

Skilled  Technical 

4.  C02  vs.  Other  JFs  (17) 

1.635 

-32.16 

1.  ST  (9) 

1.789 

— 

General  Maintenance 

2.  ST1  vs.  ST2  (17) 

2.137 

+19.45 

1 .  GM  (9) 

1.827 

-- 

3.  ST1  vs.  ST3  (17) 

1.189 

-33.54 

2.  GM1  vs.  GM2  (17) 

2.086 

+14.18 

4.  ST2  vs.  ST3  (17) 

.685 

-61.71 

3.  GM1  vs.  Other  JFs  (17) 

3.345 

+83.09 

5.  ST1  vs.  Other  JFs  (17) 

1.906 

+  6.54 

4.GM2vs.  Other  JFs  (17) 

2.096 

+14.72 

6.  ST2  vs.  Other  JFs  (17) 

3.345 

+86.98 

Mechanical  Maintenance 

7.  ST3  vs.  Other  JFs  (17) 

1.884 

+  5.31 

1 .  MM  (9) 

5.738 

- 

2.  MM1  vs.  MM2  (17) 

3.599 

-37.28 

3.  MM1  vs.  Other  JFs  (17) 

7.926 

+38.13 

4.  MM2  vs.  Other  JFs  (17) 

2.119 

-63.07 

Note.  As  in  Table  5,  distance  values  reflect  variance  of  differences  between  pairs  of  predicted 
performance  scores  multiplied  by  100.  For  each  comparison,  the  applicable  job  family  configuration  is  in 
parentheses.  %  +/-  reflects  the  percentage  change  (increase  or  decrease)  in  distance  values  going  from 
applicable  family  in  9  job  family  configuration  to  corresponding  values  representing  17  family 
configuration.  Comparisons  involving  one  job  family  versus  “other  JFs”,  excludes  related  job  families 
(i.e.,  CL1  vs.  Other  JFs,  excludes  CL2). 
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The  second  criterion  of  interest  was  whether  at  the  1 7  job  family  level  within-family 
differences  were  comparable  to  corresponding  between-family  differences.  That  is,  all  other  things 
being  equal,  it  would  be  preferable  that  differences  between  families,  on  average,  were 
substantially  larger  than  differences  within  related  families.  This  was  not  consistently  the  case 
based  on  the  current  analyses.  For  example,  differences  between  ST1,  ST2,  and  ST3  were  not 
consistently  larger  than  average  between-family  differences  involving  one  of  these  three  families 
versus  the  other  families  (in  the  17  job  family  configuration).  MM1  and  MM2  displayed  a 
comparable  trend.  Conversely,  ELI,  EL2,  and  EL3  did  exhibit  average  between-family  differences 
consistently  larger  than  corresponding  within-family  differences.  Overall,  of  the  original  9  job 
families,  only  EL  and  GM  satisfied  both  criteria  and  represented  good  candidates  for  shredding. 


As  a  follow-up  analysis  to  assess  the  practical  importance  of  these  differences,  we 
computed  the  reduction  of  “total  R-squared”  from  combining  pairs  of  job  families  from  the  17 
job  family  configuration.  By  “total  R-squared,”  we  are  referring  to  the  overall  R-squared  (or  total 
squared  composite  validity)  in  the  regression  problem  represented  by  the  combination  of  1 7 
separate  LSE  problems  corresponding  to  the  17  job  family  composites.  This  reduction  was 
computed  using  the  following  formula: 


A R} 


/  n,nj  'l 

(!) 

V*l+*7  J 

We  employed  acquisition  rather  than  observed  sample  sizes  for  the  weights  and  as  the 

former  more  appropriately  reflect  the  relative  size  of  the  job  family  in  the  Army;  n  is  the  total 
acquisition  size  across  17  job  families.  Note  that  the  expression  inside  the  first  parentheses  is  just 
the  constant  distance  statistic  formula.  The  entire  expression  above  is  a  function  of  job  family 
size  such  that  a  large  reduction  in  R2  would  be  expected  from  combining  close  to  orthogonal 
composites  (as  in  the  constant  difference  distance  statistic)  that  are  associated  with  large  job 
families.  Results  are  reported  in  Appendix  C  (see  Table  3). 

Overall,  the  pattern  was  consistent  with  those  from  the  analyses  above  of  the  variance  of 
differences  in  predicted  performance  scores.  That  is,  comparisons  previously  associated  with 
larger  variance  of  differences  in  predicted  performance  scores  were  associated  with  bigger  drops 
in  R2  even  after  taking  into  account  job  family  sizes.  Likewise,  those  families  (i.e.,  CL1,  MM1) 
that  tended  to  show  consistently  larger  variance  in  performance  score  differences  displayed 
bigger  decrements  in  R2  when  combined  with  one  of  the  other  families.  For  example,  as  with  the 
previous  analysis,  the  biggest  reduction  in  R2  across  all  possible  combinations  was  associated 
with  CL1  and  MM1  (.005819).  More  importantly,  the  loss  in  R2  from  combining  families  that 
represented  shred-outs  of  families  from  the  9  job  family  configuration  tended  to  be  small  to 
moderate,  ranging  from  .000043  (ELI  and  EL2)  to  .000928  (COl  and  C02).  The  average  loss  in 
R2  corresponding  to  these  shred-outs  came  to  .0003639,  which  loosely  corresponds  to  a  .0191 
drop  in  composite  validity  (R),  suggesting  that  the  average  loss  in  total  R2  was  not  substantially 
different  than  that  expected  by  error. 

In  summary,  the  distance  analysis  indicates  varying  differences  in  predicted  performance 
scores  between  families  at  both  the  9  and  17  job  family  configurations.  Only  a  few  families 
within  each  configuration  consistently  exhibited  relatively  sizeable  differences  in  predicted 
performance  with  the  other  families.  Comparing  differences  between  the  9  and  17  job  family 
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configurations  suggested  mixed  results.  Shredding  families  from  the  9  into  2-3  smaller  families 
for  the  17  job  family  configuration  produced  both  increments  and  decrements  in  (average) 
differential  validity  across  families.  Results  observed  when  investigating  the  expected  reduction 
in  composite  validity  (A/?2)  from  combining  families  indicate  that  these  composite  differences 
relatively  are  not  sizeable,  as  the  average  drop  in  composite  validity  (R)  was  practically  small 
when  taking  into  account  error.  As  a  whole,  these  findings  are  consistent  with  previous  research 
showing  that  test  composites,  and  corresponding  predicted  performance  scores,  tend  to  be  highly 
correlated  (Greenston  et  al„  2001).  While  the  present  findings  could  be  used  to  identify  possible 
alternative  configurations  other  than  the  proposed  9  and  17  configurations,  they  suggest  that  the 
observed  differences  are  not  likely  to  produce  practical  differences  in  aggregate  Soldier 
performance  (MPP).  The  final  joint  analysis,  summarized  next,  provided  a  more  comprehensive 
answer  to  that  question. 

Evaluating  the  Practical  Effects  of  Composite  Stability  and  Job  Family  Structure  on 
Classification  Efficiency:  An  Integrated  Analysis 

The  foregoing  analyses  demonstrated  that:  (a)  test  composites  are  reasonably  stable, 
except  for  the  150  job  configuration;  and  (b)  there  is  evidence  of  differential  validity  in  the  test 
composites,  even  when  controlling  for  sampling  error.  Although  they  suggest  implications  for 
classification  efficiency,  these  analyses  did  not  directly  measure  the  practical  effects  of  these 
issues  on  classification  efficiency.  To  assess  the  practical  effects  of  these  issues,  we  conducted  a 
comprehensive  set  of  analyses  that  modeled  random  variation  in  test  composites  induced  by 
empirically  estimating  the  weights.  Doing  this  enabled  us  to  examine  its  impact  on  both  overall 
MPP  and  differences  in  MPP  across  and  within  job  families. 

This  section  is  organized  as  follows.  First,  we  discuss  the  motivations  for  conducting  this 
analysis.  Specifically,  we  review  Zeidner  and  colleagues’  method  for  estimating  MPP  and  its 
implications  for  operational  decisions  regarding  the  proposed  composites  and  job  families. 
Second,  we  briefly  describe  the  method  used  in  the  current  analysis  and  its  advantages.  Third, 
we  summarize  our  findings  from  the  analysis. 

Estimating  MPP:  Implications  for  Operational  Decisions  Involving  the  Proposed  Composites 
and  Job  Families 

When  evaluating  which  features  of  a  classification  system  are  optimal,  it  is  strongly 
recommended  (see  Pearlman,  1980;  Sackett,  1988)  that  special  emphasis  be  placed  on  external 
criteria,  such  as  indices  of  classification  efficiency.  Consistent  with  these  recommendations,  indices, 
specifically  MPP,  have  been  used  to  inform  operational  decisions  for  structuring  Army  classification 
systems,  such  as  which  job  family  configuration  to  adopt  to  maximize  MPP  (i.e.,  Zeidner  et  al.,  2000, 
2001, 2003b).  However,  as  with  any  statistical  estimate,  the  quality  of  MPP  estimates  could  vary 
considerably  depending  on  the  estimation  procedure.  Therefore,  the  quality  of  these  estimates  is 
important,  as  their  impact  on  the  effectiveness  of  these  operational  decisions  could  be  substantial. 
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As  described  earlier,  Zeidner  and  colleagues’  have  traditionally  employed  a  double  cross- 
validation  design  for  estimating  MPP.  In  brief,  this  design  involves  deriving  the  two  sets  of 
weights  (evaluation  and  assignment)  separately  using  different  samples  of  recruits  (genetically 
referred  to  as  Samples  A  and  B,  respectively),  then  applying  both  sets  to  a  third  series  of  cross-  or 
holdout  samples  (Sample  C).  Whereas,  assignment  weights  are  used  to  classify  the  recruits,  the 
evaluation  weights  are  used  to  compute  predicted  performance  scores.  The  predicted  performance 
scores  are  then  averaged  across  recruits  within  each  cross-sample,  and  then  across  all  cross¬ 
samples  (usually  20),  to  obtain  an  estimate  of  overall  MPP.  This  overall  estimate  of  MPP  has  then 
been  used  to  evaluate  features  of  Zeidner  and  colleagues’  proposed  classification  system  or 
possible  alternatives,  such  as  the  optimal  job  family  configuration  (see  Greenston  et  al.,  2001; 
Zeidner  et  al.,  2000,  2003b).  The  purpose  of  the  double-cross  validation  design  is  to  model 
sample-to-sample  variability  (i.e.,  sampling  error)  in  MPP,  which  when  averaged  across  the 
multiple  cross-samples  is  expected  to  produce  an  unbiased  estimate  of  overall  MPP.  While  past 
research  has  been  instructive,  there  were  aspects  that  could  be  constructively  extended  to  more 
definitively  evaluate  the  classification  potential  of  the  proposed  test  composites  and  job  family 
configurations. 

First,  as  traditionally  applied,  the  double  cross-validation  design  does  not  directly  model 
error  attributable  to  the  test  composites,  and  their  respective  weights.  Under  the  double  cross- 
validation  design,  the  test  composites  are  essentially  treated  as  fixed.  That  is,  the  composite 
weights  are  treated  as  the  unknown  (“true”)  population  values,  which  by  definition  are  free  of 
error.  In  regression  terminology,  the  double  cross-validation  design  models  the  standard  error 
of  predicted  performance,  but  not  the  standard  error  of  the  composites  (and  their  respective 
weights).  While  the  two  are  related,  they  technically  are  not  the  same.  That  is,  while  standard 
errors  in  predicted  performance  are  partly  a  function  of  errors  in  the  test  composites,  they  are 
also  a  function  of  other  random  sources  of  sample-specific  variance.  Therefore,  by  excluding 
error  associated  with  the  test  composites,  the  current  design  likely  underestimates  the  level  of 
error  associated  with  MPP.  More  recent  research  by  Zeidner  and  colleagues  (Zeidner  et  al., 
2003a)  confirms  this  by  documenting  that  there  is  variability  in  predictive  validities,  and 
indirectly  the  composite  weights,  when  based  on  the  evaluation  and  assignment  samples. 

A  second  way  in  which  past  research  could  be  extended  is  that  previous  studies  have 
tended  to  focus  on  overall  MPP,  arguably  at  the  expense  of  MPP  at  the  job  family-  or  MOS- 
level.  While  overall  MPP  is  informative,  practically  the  MPP  of  the  individual  job  families  (or 
MOS)  is  expected  to  be  of  equal,  if  not  potentially  greater,  interest  to  Army  personnel  decision¬ 
makers  for  the  following  reasons.  First,  some  job  families  may  be  more  central  to  the  Army  s 
mission  than  others,  thus  decision-makers  are  likely  to  be  interested  in  MPP  estimates  for 
specific  job  families.  Second,  when  evaluating  the  proposed  composites  and  job  families, 
decision-makers  will  likewise  be  interested  in  how  MPP  is  distributed  across  the  different 
families,  such  that  the  high-performing  recruits  are  not  being  disproportionately  allocated  to 
certain  families  over  others.  Reports  of  MPP  at  the  job  family-  or  MOS-level  indicate  that  there 


16  Zeidner  and  colleagues  more  recently  introduced  a  triple  cross-validation  design  for  estimating  MPP  (see  Zeidner 
et  al.,  2003b).  Essentially,  it  is  the  same  design  as  described  previously,  except  that  participants  comprising  the 
evaluation  (Sample  A)  and  assignment  samples  (Sample  B)  are  at  one  point  switched,  so  as  to  produce  a  back-  and 
cross-sample  set  of  MPP  estimates.  The  two  sets  are  then  averaged  to  obtain  the  final  estimate  of  MPP.  This  design 
is  comparable,  but  not  equivalent,  to  the  k- fold  cross-validation  design  proposed  here,  where  k  =  2. 
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is  variability  in  MPP  across  jobs,  such  that  some  jobs  are  associated  with  negative  MPP  values, 
whereas  others  with  positive  MPP  values  (Zeidner  et  al.,  2000,  2003b).  Third,  external  criteria, 
such  as  MPP,  can  be  instructive  when  assessing  job  similarity  for  purposes  of  determining 
differential  validity.  That  is,  if  two  job  families  are  comparable  in  their  composites,  differences 
in  MPP  can  be  informative  for  externally  validating  the  practical  significance  of  these 
differences.  Because  some  job  family-level  MPPs  will  be  based  on  smaller  sample  sizes  (n), 
even  when  aggregated  across  multiple  cross-samples,  the  standard  error  of  MPP  for  these 
families  will  be  higher  than  that  for  all  families  (as  a  whole).  Therefore,  this  error  could 
substantially  impact  conclusions  about  between-family  differences  in  MPP. 

In  summary,  while  previous  research  has  its  strengths  and  has  been  informative,  there  are 
limitations  that  could  impact  operational  decisions  based  on  these  estimates.  Specifically,  these 
limitations  are:  (a)  the  double  cross-validation  design  typically  employed  in  past  studies  does  not 
model  error  in  the  test  composites,  and  their  respective  weights,  thereby  likely  underestimating 
error  in  MPP;  and  (b)  past  research  tends  to  focus  on  overall  MPP  with  less  attention  to  MPP  at 
the  job  family-  or  MOS-level.  To  address  these  issues,  we  conducted  a  constructive  simulation 
to  model  the  practical  effects  of  estimation  error  in  the  composites  on  MPP  and  its  implications 
for  optimizing  classification  using  the  proposed  job  families. 

Method 

For  the  current  analyses,  we  conducted  a  constructive  simulation  with  multiple 
replications  using  actual  ASVAB  and  performance  data  from  the  large-scale  SQT  database  from 
which  the  proposed  composites  and  job  family  configurations  were  derived.  The  design  of  the 
simulation  closely  represents  an  extension  of  Zeidner  and  colleagues’  double-cross  validation 
design,  and  involved  the  following  steps. 

First,  similar  to  Zeidner  and  colleagues’  design,  we  randomly  assigned  individual  recruits 
to  one  of  three  types  of  samples.  Specifically,  we  assigned  a  subset  of  the  total  sample 
(n  =  5,000)  to  one  of  5  cross-samples  of  1,000  each  (Sample  C),  then  equally  partitioned  the 
remaining  recruits  ( n  —  250,000)  into  an  assignment  (Sample  A)  and  an  evaluation  sample 
(Sample  B).  For  the  second  step,  we  estimated  the  applicable  AA  and  PP  composites  using  the 
assignment  and  evaluation  samples.18  Third,  we  optimized  the  classification  of  recruits  in  each 
of  the  cross-samples  based  on  scores  computed  using  the  previously  derived  AA  composites  and 
the  same  allocation  percentages  reported  in  Zeidner  and  colleagues’  previous  work  (see  Zeidner 
et  al.,  2001).  Fourth,  after  assignment,  we  computed  predicted  performance  scores  for  each 
participant  in  the  cross-samples  using  the  PP  composites,  likewise  previously  derived.  As  with 
Zeidner  and  colleagues,  to  obtain  mean  predicted  performance  (MPP)  we  averaged  PP  scores 
across  participants  within  each  cross-sample.  Fifth,  and  finally,  we  repeated  the  first  four  steps 
49  times  to  obtain  data  for  49  replications  ( k  =  49).  To  ensure  that  the  majority  of  the 
participants  in  the  total  sample  contributed  data  to  the  cross-samples  (Sample  C),  we  initially 


17  Consistent  with  Zeidner  and  colleagues,  AA  and  PP  composite  weights  were  based  on  ASVAB-SQT  validities 
corrected  for  criterion  unreliability  and  range  restriction.  As  we  were  interested  in  the  contributions  of  classification 
(and  not  selection)  to  MPP,  validities  were  corrected  to  the  Army  Input  population,  which  represents  all  recruits 
qualified  to  serve  in  the  Army  and  eligible  for  assignment  to  entry-level  MOS. 
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partitioned  the  entire  data  into  49  subsamples  of  5,000  each.19  For  each  of  the  49  replications, 
one  of  the  49  subsamples  served  as  Sample  C.  The  first  four  steps  are  consistent  with  Zeidner 
and  colleagues’  double  cross-validation  design.  The  fifth  step,  replicating  the  double  cross- 
validation  design  k  times,  extends  Zeidner  and  colleagues’  design,  enabling  us  to:  (a)  directly 
model  random  variation  (error)  in  estimating  the  AA  and  PP  composite  weights;  and  (b)  evaluate 
the  practical  effects  of  this  variation  when  making  comparisons  involving  MPP.20 

To  assess  differences  by  job  family  configuration  and  composite  weight  derivation,  we 
employed  a  3  x  2  x  2  design.  That  is,  we  repeated  the  above  design  10  times  to  obtain  MPP 
estimates  to  evaluate  the  comparisons  of  interests.  There  was  one  factor  reflecting  job  family 
configuration  and  two  factors  reflecting  differences  in  how  the  assignment  composite  weights 
were  estimated.  The  job  family  configuration  factor  had  three  levels  (9,  17,  and  150)  reflecting 
the  alternative  job  family  configurations  for  the  “operational”  two-tiered  classification  system. 

As  for  the  two  composite  estimation  factors,  one  focused  on  the  type  of  constraint  placed  when 
estimating  the  weights  and  consisted  of  two  levels:  no  constraint  (i.e.,  observed  OLS  weights) 
versus  positive  constraint  (i.e.,  weights  are  constrained  to  be  positive).  The  second  estimation 
factor  dealt  with  the  impact  of  standardizing  the  weights  to  produce  scores  with  equal  mean  and 
variance,  and  was  comprised  of  two  levels:  unstandardized  versus  standardized.  Consistent  with 
its  “operational”  implementation,  the  evaluation  weights  used  in  computing  individual  predicted 
performance  scores  were  always  derived  using  the  150  job  family  configuration  and 
unconstrained,  OLS-regression  weights.  Since  composite  weights  for  the  150  job  family 
configuration  are  not  constrained  to  be  positive,  only  the  no  constraint  condition  was  relevant 
when  estimating  MPP  for  the  150  configuration.  The  conditions  comprising  the  design  are 
summarized  in  Table  7  below.  The  SAS  programs  for  replicating  the  design,  with  accompanying 


documentation,  are  found  in  Appendix  D. 

Table  7 

Summary  of  Conditions  in  3x2x2  Design 

Constraint 

Standardization 

No  Constraint 

Positive  Constraint 

Unstandardized 

■  9, 17,  and  150  JF 
Configurations 

■  9  and  17  JF 
Configurations 

Standardized 

■  9,  17,  and  150  JF 
Configurations 

■  9  and  17  JF 
Configurations 

This  design  offers  several  advantages.  First,  as  with  Zeidner  and  colleagues,  it  places 
emphasis  on  the  practical  effects  of  composite  stability  and  job  family  configuration,  specifically 
classification  efficiency  (i.e.,  MPP),  and  not  strictly  internal  or  statistical  criteria.  Second,  the 
current  design  extends  Zeidner  and  colleagues’  work  by  directly  modeling  error  in  the  composites, 
and  their  respective  weights;  this  error  is  expected  to  influence  estimates  of  MPP .  By  taking  into 
account  error  in  the  composites,  we  can  more  accurately  evaluate  differences  in  MPP  owing  to 


A  50th  subsample  with  n  less  than  5,000  was  also  derived,  which  consisted  of  the  n  recruits  remaining  after 
partitioning  data  into  the  first  49  subsamples.  This  50th  subsample  was  not  used  in  our  design. 

'9  This  method  is  also  known  as  fc-fold  (double)  cross-validation. 


24 


different  job  family  configurations  and  other  proposed  features  of  a  classification  system,  so  as  to 
more  confidently  inform  operational  choices  about  Army  classification  policy.  Third,  the  design 
extends  Zeidner  and  colleagues’  research  by  considering  MPP  estimates  at  the  job  family-level. 

Summary  of  Results 

As  discussed  earlier,  the  current  analyses  were  meant  to  assess  the  impact  of  job  family 
configuration  and  composite  estimation  factors  on  estimates  of  overall  MPP  and  differences  in 
MPP  both  across  and  within  job  families,  taking  into  account  composite  stability.  We  turn  to  a 
summary  of  the  major  findings  for  each  in  turn. 

Overall  MPP.  To  evaluate  the  effects  of  job  family  configuration  and  composite 
estimation  factors  on  overall  MPP,  we  obtained  estimates  of  overall  MPP  using  the 
aforementioned  procedure.  These  estimates  were  then  analyzed  using  a  standard  analysis  of 
variance  (ANOVA)  with  MPP  as  the  dependent  variable,  and  the  independent  variables  being  the 
three  factors  comprising  our  design  (with  abbreviations  in  parentheses):  (a)  job  family 
configuration  (JFCONFIG);  (b)  no  constraint  vs.  positive  constraint  (CONSTRAINT);  and  (c) 
unstandardized  vs.  standardized  weights  (STAND).  Results  from  this  ANOVA  are  summarized 
in  Table  8.  From  the  results,  we  observed  the  following. 

First,  neither  the  main  effect  nor  interactions  involving  CONSTRAINT  were  statistically 
significant  (p  >  .05).  Similarly,  associated  effect  sizes  (R2)  were  zero.  Therefore,  even  without 
taking  into  account  estimation  error,  constraining  weights  to  be  positive  did  not  materially  affect 
MPP.  Second,  the  main  effects  involving  STAND  (F(i,2450)  ~  7459.934 ,p  <  .0005)  and  job 
configuration  (F(2,2450)  =  1867.287,/?  <  .0005)  were  statistically  significant.  Because  of  the  large 
sample  size  (N),  we  also  computed  effect  size  estimates.  Both  exhibited  generally  large  effect 
sizes,  with  STAND  (partial  k  =  .754,  R2  =  .517)  explaining  more  of  the  variability  in  MPP  than 
JFCONFIG  (partial  R2  =  .605,  R2  =  .259).  Therefore,  both  standardization  of  weights  and  job 
configuration  significantly  contributed  to  differences  in  MPP.  Third,  and  finally,  there  was  a 
significant  interaction  between  STAND  and  JFCONFIG  (F(2,2450) =  212.346,/?  <  .0005),  although 
the  magnitude  of  this  effect  was  noticeably  smaller  relative  to  its  component  main  effects  (partial 
R2  =  .148 ,R2  =  .030).  Therefore,  differences  in  MPP  by  job  configuration  were  dependent  on 
whether  the  weights  were  standardized  or  not.  We  investigated  the  nature  of  this  effect  and  the 
above  main  effects  more  fully,  including  the  implications  of  estimation  error,  in  a  series  of 
follow-up  analyses. 

We  followed-up  the  omnibus  ANOVA  with  individual  ANOVAs  designed  to  test  for 
simple  effects,  with  an  emphasis  on  job  family  configuration.  These  ANOVAs  followed  the 
omnibus  model  described  previously,  except  there  was  a  single  independent  variable  (job  family 
configuration)  and  each  focused  on  MPP  estimates  obtained  for  a  specific  condition.  As  there 
were  4  conditions  total  (excluding  job  configuration),  we  conducted  4  ANOVAs.  Results  from 
these  analyses,  including  estimates  of  standard  error  of  MPP,  are  summarized  in  Table  9. 
Consistent  with  the  significant  main  effect  observed  previously  for  standardization  of  the 
weights,  MPP  estimates  tended  to  be  systematically  lower  when  weights  were  standardized. 
Evidence  for  the  significant  interaction  between  standardization  and  job  configuration  can  be 
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Table  8 

Results  of  Analysis  of  Variance  (ANOVA)  of  Overall  MPP 


df _ Partial  R2 


JF  Configuration  (JF  CONFIG) 

1867.287* 

2,2450 

.605 

.259 

Constraint  (CONT) 

.084 

1,2450 

.000 

.000 

Standardization  (STAND) 

7459.934* 

1,2450 

.754 

.517 

JF  CONFIG  x  CONT 

.000 

1,2450 

.000 

.000 

JF  CONFIG  x  STAND 

212.346* 

2,2450 

.148 

.030 

JF  CONFIG  x  CONT  x  STAND 

.004 

2,2450 

.000 

.000 

Overall  R 2 

.831 

IVUlt y  ^  .UUUJ .  x  autai  ii  iv^iv^v»n.u  — - —  c  ± 

partialed  out  variance  attributable  to  the  other  factors.  R  represents  percentage  of  total  variance  in 
weights  explained  by  factor. 


seen  in  that  the  differences  in  MPP  by  standardization  increased  as  the  number  of  jobs  increased. 
For  example,  looking  at  the  9  job  family  configuration,  standardizing  the  weights  produced  MPP 
values  of  roughly  .08  versus  .12  when  weights  were  unstandardized.  When  moving  to  the  17  job 
family  configuration,  the  difference  widened,  as  standardizing  the  weights  produced  MPP  values 
of  .09,  whereas  unstandardized  weights  resulted  in  MPP  values  around  .14,  roughly  a  25% 
increase  in  the  difference.  Increasing  the  number  of  jobs  almost  ten-fold  to  150  increased  the 
difference,  with  standardized  weights  producing  an  MPP  of  .12  versus  .19  for  unstandardized 
weights  —  a  40%  increase  in  the  difference  over  the  17.  In  sum,  the  magnitude  of  the  difference 
in  overall  MPP  between  standardized  and  unstandardized  weights  varied  partly  as  a  function  of 
job  configuration.  A  possible  explanation,  with  implications  for  what  this  means  operationally, 
are  discussed  shortly. 


As  for  job  family  configuration,  MPP  significantly  differed  by  configuration  across  all 
conditions.  As  evident  from  Table  10,  differences  in  MPP  among  the  three  job  configurations 
were  statistically  significant  (p  <  .0005).  The  magnitude  of  the  corresponding  effect  sizes  (R  ) 
varied,  ranging  from  low  (.040)  to  high  (.801).  A  closer  inspection  of  the  differences  showed 
that  overall  MPP  generally  increased  as  the  number  of  job  families  increased,  although  the  rate  at 
which  MPP  improved  markedly  declined  with  the  added  number  of  job  families.  For  example, 
when  looking  at  the  no  constraint  -  standardized  condition,  MPP  goes  from  .083875  to  .089514 
(a  6.72%  increase)  when  the  number  of  jobs  are  doubled  (from  9  to  17),  and  to  .119391  (a  33.4% 
increase)  when  the  number  of  jobs  increases  almost  ten- fold  from  17  to  150.  Consistent  with  the 
significant  interaction  between  job  configuration  and  standardization,  these  differences  varied  by 
whether  the  weights  were  standardized  or  not.  In  contrast  to  the  preceding  example,  for  the  no 
constraint  -  unstandardized  condition,  MPP  increased  from  .1 19569  to  .140396  (a  17.4% 
increase)  when  the  number  of  jobs  doubled  (from  9  to  17),  and  to  .186598  (a  32.9%  increase) 
when  increasing  the  number  of  jobs  from  17  to  150.  In  sum,  while  overall  MPP  differed  by  job 
family  configuration,  such  that  MPP  generally  increased  as  the  number  of  jobs  increased,  the  rate 
at  which  MPP  improved  steadily  declined  with  the  added  number  of  jobs.  In  addition,  the 
magnitude  of  these  differences  and  rate  of  improvement  in  MPP  depended  on  whether  weights 
were  standardized  or  not. 
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Several  important  observations  are  of  note.  First,  the  magnitude  of  the  standard  errors  of 
MPP  observed  using  the  current  design  tended  to  be  larger  than  those  previously  reported  by 
Zeidner  and  colleagues  (Zeidner  et  al.,  2000,  2003b).  Although  comparison  values  were  not 
readily  available  for  all  conditions,  the  estimates  produced  here  were  generally  25%  larger  than 
those  previously  reported.  These  data  suggest  that  test  composite  stability  could  meaningfully 
influence  operational  decisions,  based  on  MPP,  given  the  relatively  small  differences  in  MPP 
observed  between  many  of  the  job  family  configurations  and  across  conditions. 

Second,  and  related  to  the  first  point,  taking  into  account  the  standard  error  of  MPP,  there 
was  sizeable  overlap  in  MPP  across  the  job  family  configurations  (see  Table  10).  In  particular, 
there  was  considerable  overlap  in  overall  MPP  when  weights  were  standardized  and  when 
comparing  the  9  to  the  17  job  family  configurations.  For  example,  looking  at  the  no  constraint  - 
standardized  condition,  there  was  substantial  overlap  in  the  confidence  intervals  for  MPP  for  the 
9, 17,  and,  to  a  lesser  extent,  150  job  family  configurations.  Practically,  this  confirmed  that 
statistically  significant  omnibus  differences  in  MPP  by  job  family  configuration  reported 
previously  were  mainly  due  to  differences  between  the  9  and  150  job  families. 

A  third  and  final  observation  pertains  to  the  finding  that  while  overall  MPP  improved  as 
the  number  of  jobs  increased,  the  magnitude  of  these  increases  diminished  in  relation  to  the 
number  of  jobs.  For  example,  as  reported  above,  doubling  the  number  of  jobs  from  9  to  17 
tended  to  produce  a  jump  in  overall  MPP  of  7%  to  1 7%.  By  comparison,  increasing  the  number 
of  jobs  ten-fold  at  best  increased  MPP  by  one-third  (33%  total).  This  suggests  that  most  of  the 
differentiation  among  jobs,  given  the  population  of  Army  jobs  from  which  the  composites  and 
job  families  were  constructed,  is  attributable  to  the  9  and/or  17  job  family  configurations.  These 
results  support  earlier  analyses  showing  greater  between- job  than  within-job  differences  for  the  9 
and  17  job  families. 

In  summary,  when  taking  into  account  estimation  error  in  test  composites  and  their 
respective  weights,  there  appears  to  be  no  practical  difference  in  overall  MPP  between  the  9  and  17 
job  family  configurations,  and  to  a  lesser  extent,  the  17  and  150  configurations.  This  is  particularly 
the  case  when  test  composites  are  based  on  standardized  weights.  Specifically,  when  weights  are 
standardized  to  produce  predicted  assignment  (AA)  scores  with  equal  means  and  variances, 
observed  differences  in  MPP  were  generally  small  and  within  the  standard  error  of  MPP,  especially 
when  comparing  the  9  to  the  17  job  family  configurations  (.083988  versus  .089706).  This  finding 
is  significant  for  two  reasons.  First,  from  a  practical  perspective,  MPP  estimates  based  on 
standardized  weights  more  closely  satisfy  existing  operational  constraints,  as  they  produce  a  more 
equitable  distribution  of  recruit  quality  across  job  families.  Thus,  these  weights  mimic  statistically 
the  distributional  requirements  current  Army  classification  policy  considers  when  classifying 
recruits  to  entry-level  MOS. 

A  second  reason  for  the  significance  of  this  finding  is  that  it  suggests  that  specific 
abilities  and  aptitudes  contribute  less  to  classification  efficiency  relative  to  general  mental  ability 
(GMA),  given  the  current  population  of  jobs.  We  are  able  to  infer  this  because  standardizing  the 
weights  essentially  equates  the  composites  in  terms  of  GMA,  such  that  what  remains  after 
standardization  reflects  the  unique  contributions  of  specific  abilities  and  aptitudes  to 
classification  independent  of  GMA.  According  to  DAT,  increasing  the  number  of  jobs  should 
maximize  differences  among  jobs  in  their  performance  requirements,  thereby  increasing  the 
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classification  efficiency  of  a  multidimensional  test  battery  and  corresponding  composites  based 
on  specific  aptitudes.  As  reported  previously,  larger  MPP  values  were  observed  when  weights 
were  unstandardized,  and  the  increases  in  MPP  as  the  number  of  jobs  increased  (from  9  to  17  to 
150)  were  greater  than  with  standardized  weights.  These  increases  in  MPP  are  generally 
attributable  to  higher  predictive  (or  criterion-related)  validities,  which  are  largely  a  function  of 
GMA  (Hunter,  1983;  Ree  &  Earles,  1991;  Ree,  Earles,  &  Teachout,  1994).  When  these 
validities  were  effectively  equated  by  standardizing  the  weights,  both  the  magnitude  of  MPP  and 
increases  in  MPP  from  moving  to  9  to  17  to  150  jobs  declined  appreciably.  Therefore,  the 
decline  in  MPP  improvement  as  the  number  of  jobs  increases,  particularly  when  the  number  of 
jobs  is  increased  ten-fold,  indicates  that  a  substantial  portion  of  the  pre-standardization 
differentiation  (i.e.,  differential  validity)  among  jobs  is  attributable  to  GMA,  not  specific  abilities 
and  aptitudes. 

MPP  by  Job.  To  evaluate  the  effects  of  job  family  configuration  and  estimation  method 
on  MPP  at  the  job-level,  we  computed  estimates  of  MPP  for  each  job  using  the  recorded  job- 
level  assignments  and  predicted  performance  scores  from  the  preceding  simulation.  As  there 
was  no  evidence  from  the  above  analyses  that  constraining  weights  to  be  positive  materially 
affected  MPP  estimates,  this  factor  was  dropped  from  the  design.  We  used  the  150  MOS 
comprising  the  150  job  family  configuration  to  define  the  job-levels.  This  enabled  us  to 
investigate  how  the  proposed  test  composites  would  function  operationally,  as  this  is  the  level  at 
which  actual  classification  decisions  are  made  in  the  field,  and  not  at  an  aggregate  level,  such  as 
the  9  or  17  job  families.  Therefore,  we  obtained  estimates  of  MPP  for  150  jobs  using  the  AA 
assignment  composites  based  on  the  9, 17,  and  150  job  family  configurations.  To  assess 
differences  in  composite  estimation  method,  one  set  of  MPP  estimates  was  based  on 
unstandardized  AA  weights  and  the  second  set  on  standardized  AA  weights.  To  facilitate  their 
interpretation  and  the  identification  of  major  trends,  results  are  represented  visually  in  Figures  1 
to  6  (Appendix  E).21  Tables  containing  MPPs  and  standard  errors  by  job  family  configuration 
and  standardization  can  be  found  in  Appendix  F.  From  these  figures  and  tables,  we  identified 
several  major  trends. 

First,  as  to  be  expected,  and  consistent  with  previous  results  (Zeidner  et  al.,  2000,  2003b), 
the  level  of  MPP  differed  by  job.  However,  the  magnitude  and  pattern  of  these  differences 
noticeably  varied,  depending  on  whether  weights  were  standardized  or  not.  Specifically,  there 
tended  to  be  greater  between-job  differences  in  MPP  when  weights  were  unstandardized  than 
standardized,  such  that  some  jobs  were  associated  with  low,  and  in  some  cases,  negative  MPP 


20  Each  box-and- whisker  plot  in  these  figures  represents  the  distribution  of  MPP  for  a  specific  job  over  245  cross¬ 
samples,  given  a  particular  job  family  configuration  (9,  17,  or  150).  The  dot  inside  the  box  represents  the  median  of 
the  distribution.  The  width  of  the  box  represents  the  inter-quartile  range  (IQR)  or  middle  50%  of  the  distribution. 
The  whiskers  extend  to  1.5  IQR  from  the  median  or  to  the  most  extreme  MPP  in  the  distribution,  whichever  is  most 
applicable. 
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values,  whereas  others  produced  strongly  positive  MPP  values.  For  example,  looking  at  Figure 
1,  under  the  9  job  family  configuration,  CO  jobs  were  consistently  associated  with  negative  MPP 
values,  whereas  EL  jobs  were  associated  with  positive  MPP  values.  Moving  to  the  17  job  family 
configuration,  one  observes  a  comparable  pattern,  as  C02  jobs  tended  to  produce  MPP  values 
close  to  or  slightly  greater  than  zero,  whereas  COl  jobs  continued  to  attract  low  performers, 
resulting  in  negative  MPP  values.  At  the  150  job  family  configuration,  while  there  was  less 
consistency  in  MPP  for  families  as  a  whole  (as  the  families  have  been  shredded  into  their 
constituent  jobs),  there  was  even  greater  between-job  variability  in  MPP,  with  particular  jobs 
(i.e.,  71D)  clearly  benefiting  over  others  (i.e.,  1  IB).  These  findings  are  consistent  with  research 
by  Zeidner  and  colleagues  at  the  job  family-level  (Zeidner  et  al.,  2000,  2003b). 

In  contrast,  when  looking  at  MPP  based  on  standardized  weights,  there  was  substantially 
less  variability  in  MPP  across  jobs.  That  is,  unlike  unstandardized  weights,  we  did  not  observe 
jobs  with  relatively  large  positive  MPP  achieved  at  the  expense  of  other  jobs,  as  indicated  by 
comparatively  lower  and/or  negative  MPPs.  For  example,  as  evident  from  Figure  4,  MPP  values 
for  CO  and  EL  jobs  were  generally  comparable.  Of  particular  interest  given  its  centrality  to  the 
Army’s  mission,  MPP  for  CO  jobs  was  higher,  on  average,  than  MPP  produced  when  using 
unstandardized  weights.  As  with  the  unstandardized  weights,  while  differences  in  MPP  across 
jobs  increased  as  the  number  of  job  families  increased,  from  the  9  to  the  17  to  the  150  job  family 
configurations,  the  magnitude  of  these  differences  were  smaller  relative  to  the  same  differences 
observed  using  the  unstandardized  weights. 

Second,  comparable  to  results  for  overall  MPP,  differences  in  job-level  MPP  by  job 
family  configuration  tended  to  be  small  when  taking  into  account  standard  error.  Therefore,  it 
was  difficult  to  conclude  that  one  configuration  produced  a  substantially  higher  level  of  MPP 
than  an  alternative  configuration,  particularly  when  comparing  the  9  and  17  configurations. 
While  at  the  aggregate  level,  the  magnitude  of  the  standard  error  for  the  same  job  across  the  9, 

17,  and  150  job  configurations  tended  to  be  equivalent,  there  were  sizeable  overlaps  in  job-level 
MPP  by  configuration.  For  example,  Combat  (CO)  jobs,  such  as  1  IB,  1 1C,  and  1 1H,  displayed 
relatively  small  standard  errors  (on  average).  However,  when  comparing  MPP  for  these  jobs 
across  the  three  job  family  configurations  (9, 1 7, 1 50),  the  observed  differences  in  MPP  by 
configuration  were  practically  small,  when  factoring  in  the  applicable  standard  errors.  Similarly, 
where  there  were  sizeable  differences  in  MPP,  they  were  primarily  between  the  9  and  150 
configurations  and  not  the  9  and  1 7  job  family  configurations. 

In  summary,  results  at  the  job-level  were  generally  consistent  with  those  observed  for 
overall  MPP.  Both  the  magnitude  of  MPP  and  between-job  differences  in  MPP  were,  on 
average,  contingent  on  job  configuration  and  standardization  of  the  weights.  Specifically,  job- 
level  MPP  tended  to  be  higher  when  weights  were  unstandardized  than  standardized.  In 
addition,  between-job  differences  in  MPP  were  greater  between  the  9  and  1 50  job  family 
configurations  than  between  the  9  and  1 7  configurations,  particularly  when  taking  into  account 
the  standard  error  of  MPP. 

As  before,  these  findings  suggest  that  standardized  weights  are  preferable  to 
unstandardized  weights  for  drawing  research-based  conclusions  about  the  efficacy  of  proposed 
classification  system  features  and  when  making  operational  classification  decisions.  For  one, 
standardized  weights  produce  less  variability  in  MPP  across  jobs,  ensuring  a  more  equitable 
distribution  of  recruit  quality  while  at  the  same  time  yielding  overall  Army  classification 
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benefits.  Results  based  on  standardized  weights  indicate  that  classification  benefits  could  be 
achieved  in  a  manner  that  is  consistent  with  Army  distributional  requirements.  A  second 
advantage  of  standardized  weights  is  that  they  produce  significantly  fewer  jobs  exhibiting 
negative  MPP  values;  that  is,  fewer  jobs  where  average  aggregate  performance  is  expected  to  be 
negative.  In  particular,  for  certain  jobs  that  are  especially  critical  to  the  Army’s  mission  (i.e., 
CO),  using  standardized  weights  for  classification  resulted  in  higher  MPP  compared  to  the  MPP 
obtained  using  unstandardized  weights. 

Taken  together,  these  findings  confirm  that  standardized  weights  are  more  likely  to 
maximize  aggregate  performance,  while  ensuring  an  equitable  distribution  of  recruit  quality 
across  jobs.  By  approximating  Army  distributional  requirements,  assignments  based  on 
standardized  weights  should  theoretically  require  fewer  ad-hoc  adjustments  by  Army  personnel 
managers.  This  is  beneficial  because  such  adjustments,  which  will  likely  be  unsystematic,  would 
otherwise  negate  the  classification  efficiency  of  the  proposed  regression- weighted  test 
composites. 


Discussion 

As  noted  by  Pearlman  (1980),  job  classification  research  involves  both  a  process  and  a 
product.  The  purpose  of  the  current  report  was  to  evaluate  the  process  and  product  of  the  proposed 
AA  test  composites  and  their  efficacy  in  Army  classification.  In  this  case,  process  refers  to  how 
the  AA  composites  were  derived,  whereas  product  references  the  stability  and  classification 
potential  of  the  composites.  This  section  is  organized  as  follows.  First,  we  summarize  the  major 
findings  from  our  evaluation  of  the  test  composites  and  we  review  outstanding  implementation 
issues.  Second,  we  make  recommendations  regarding  the  adoption  of  the  proposed  AA 
composites.  Third,  and  finally,  we  offer  several  suggestions  for  future  research  to  further  improve 
Army  classification  and  selection. 

Summary  of  Major  Findings 

Our  evaluation  focused  on  investigating  the  stability  and  differential  validity  associated 
with  the  proposed  regression-weighted  AA  composites,  and  their  practical  effects  on 
classification  efficiency  (as  measured  by  MPP).  In  particular,  we  focus  on  differences  between 
the  9  AA  composites  and  an  alternative  set  of  17  composites,  which  represent  different  versions 
of  the  assignment  tier  in  the  “operational”  version  of  Zeidner  and  colleagues’  proposed  two- 
tiered  classification  system.  Overall,  we  found  the  following: 

First,  regression-weighted  test  composites  for  the  9  and  17  job  family  configurations 
demonstrated  the  greatest  stability.  Both  the  composites  as  a  whole,  and  their  respective 
weights,  displayed  smaller  standard  errors  and  fewer  problems  (i.e.,  collinearity,  “weak”  data) 
than  those  for  the  150  job  family  configuration.  Based  on  our  findings,  the  stability  of  the  test 
composites  comprising  the  150  job  family  for  use  in  assigning  recruits  to  entry-level  MOS 
configuration  is  questionable.  This  is  because  the  lower  n,  on  average,  on  which  estimates  are 
based,  is  associated  with  higher  standard  errors  and  greater  susceptibility  to  inflation  due  to 
multicollinearity  present  among  the  ASVAB  subtests.  There  is  no  compelling  evidence  that  the 
9  are  necessarily  more  stable  than  the  17,  as  both  are  based  on  families  of  relatively  large  n  and 
the  effects  of  multicollinearity  and  “weak”  data  are  comparable  across  the  two  sets  of 
composites. 
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Second,  as  currently  constructed,  the  9  and  1 7  AA  composites  demonstrated  between- 
family  differences  in  validities,  even  when  taking  into  account  sampling  error.  Specifically,  both 
the  9  and  1 7  AA  composites  displayed  systematic  between-family  differences  in:  (a)  composite 
validities  (Rs),  operationalized  as  differences  in  their  respective  weights,  both  individually  and  as  a 
full  set;  and  (b)  predicted  performance  scores  (r).  However,  these  differences  varied  by  job  family, 
such  that  some  families  (i.e.,  Clerical  and  Mechanical  Maintenance)  within  each  set  of  composites 
were  more  consistently  and  strongly  differentiated  from  the  other  jobs  than  others.  Equally  as 
important,  shredding-out  families  and  composites  at  the  9  job  family  configuration  to  produce  the 
1 7  configuration  resulted  in  mixed  findings.  Whereas  some  jobs  displayed  greater  differentiation 
in  validities  and  predicted  performance  scores,  other  jobs  did  not.  Therefore,  the  gains  in 
differential  validity  were  not  uniform.  Equally  as  important,  they  were  partially  offset  by  losses  in 
differentiation  for  other  jobs.  Overall,  moving  to  the  17  AA  composites  does  not  appear  to 
produce  greater  levels  of  differential  validity  than  that  observed  with  the  9  AA  composites. 

Third,  and  most  importantly,  differences  in  mean  predicted  performance  (MPP)  among 
the  9,  17,  and  150  AA  test  composites  were  not  practically  significant,  especially  after  taking 
into  account  estimation  error  in  MPP.  That  is,  both  overall  MPP  and  job-level  MPP  did  not 
substantially  differ  across  job  family  configuration,  particularly  between  the  9  and  17  AA 
composites,  after  considering  variability  in  MPP  due  to  estimation  error.  The  finding  that  there 
is  no  practical  difference  in  MPP  between  the  9  and  17  AA  composite  is  consistent  with  recent 
research  (Zeidner  et  al.,  2003b).  In  addition,  we  considered  the  practical  effects  on  MPP  of 
several  operational  issues  related  to  how  test  composites,  and  their  respective  weights,  are 
estimated.  Whereas,  constraining  weights  to  be  positive  did  not  meaningfully  impact  MPP, 
standardizing  predicted  performance  scores  to  have  equal  means  and  variance  did.  Specifically, 
standardized  weights  produced  somewhat  lower  (on  average)  overall  MPP  than  unstandardized 
weights,  but  a  more  equitable  distribution  in  MPP  across  jobs  and,  in  some  cases,  higher  MPP 
for  jobs  (i.e.,  Combat)  critical  to  the  Army’s  mission.  Because  it  already  integrates  important 
operational  constraints,  a  classification  system  based  on  standardized  weights  will  require  fewer 
interventions  that  might  otherwise  negate  the  intended  benefits  of  regression-weighted 
composites. 

In  considering  outstanding  implementation  issues,  the  most  prominent  issue  pertains  to 
cut  scores  for  the  17  and  150  AA  composites.  We  estimated  cut  scores  for  the  17  and  150  AA 
composites,  equating  the  new  cut  scores  with  cut  scores  for  the  9  AA  composites,  so  as  to 
produce  comparable  MPP  values.  Cut  scores  and  overall  selection  ratio  by  MOS  based  on  the  17 
and  150  AA  composites,  with  accompanying  documentation  on  our  cut  score  equating  method, 
are  reported  in  Appendix  G.  As  can  be  seen  from  these  data,  moving  to  the  17  or  150  AA 
composites  would  produce  changes  in  cut  scores.  While  these  changes  are  generally  small,  1-3 
points,  even  small  changes  could  be  associated  with  substantial  costs  given  the  number  of  MOS 
affected.  More  importantly,  the  selection  ratios  (percentage  of  recruits  qualifying  for  an  MOS) 
likewise  change.  For  example,  at  the  17  job  family  levels,  the  change  in  percentage  qualifying 
ranges  from  less  than  1%  to  3%,  whereas  with  the  150  job  family  level  the  percentage  change 
ranges  from  less  than  1%  to  18%.  To  ensure  comparable  fill-rates  with  the  new  cut  scores  would 
necessitate  potentially  even  larger  changes  in  cut  scores.  Therefore,  adoption  of  the  17  or  150 
AA  composites  for  classification  would  necessitate  substantive  changes  in  cut  scores  to  achieve 
comparable  levels  of  MPP  or  fill-rates. 
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Recommendations 


Based  on  the  above  findings,  we  recommend  adoption  of  the  9  AA  composites  based  on 
standardized  weights  for  use  in  assigning  recruits  to  entry-level  MOS.  There  are  two  reasons  for 
this.  First,  consistent  with  previous  research  (Zeidner  et  al.,  2003b),  moving  from  9  to  17  AA 
composites  does  not  produce  a  practically  significant  increase  in  either  overall  MPP  or  MPP  by 
MOS.  This  is  particularly  true  once  one  takes  into  account  estimation  error  in  MPP.  Coupled 
with  the  administrative  costs  and  other  management-related  issues  associated  with  changing 
existing  cut  scores  based  on  the  9  composites,  the  technical  and/or  practical  advantages  to 
adopting  the  17  AA  test  composites,  as  currently  constructed,  are  few.  The  same  holds  for  the 
150  AA  test  composites,  whose  weights  are  considerably  less  stable  and  reliable  than  the  9  or  17 
composites  and,  thereby,  less  technically  defensible  and  feasible  for  supporting  assignment 
decisions. 

The  second  advantage  to  adopting  the  9  AA  composites  based  on  standardized  weights  is 
that  they  promote  a  more  equitable  distribution  of  MPP  across  MOS  than  unstandardized 
composites.  Practically,  this  means  that  lower  quality  recruits  are  less  likely  to  be 
disproportionately  assigned  to  selected  MOS,  specifically  those  with  lower  ASVAB  predictive 
validities,  and  vice  versa  with  higher  quality  recruits.  Standardized  composites  accomplish  this 
by  statistically  approximating  critical  operational  requirements,  such  as  MOS  distributional 
requirements,  when  assigning  recruits  to  MOS.  In  doing  so,  standardized  composites  retain  the 
classification  efficiency  of  regression-weighted  composites,  while  taking  into  account  practical 
operational  concerns  that  might  otherwise  reduce  their  benefits.  If  unstandardized  composites 
were  used,  Army  classification  managers  and  other  decision-makers  would  have  to  make  greater 
adjustments  to  assignments  based  on  regression- weighted  composites  in  order  to  achieve 
practical  objectives  (i.e.,  distributional  requirements,  fill-rates).  Since  these  adjustments  are 
discretionary  and  not  likely  to  be  implemented  systematically  across  recruits  and/or  MOS,  they 
could  be  expected  to  negate  the  classification  potential  of  the  9  (standardized)  composites.  In 
sum,  standardized  composites  will  more  effectively  balance  the  optimization  of  aggregate 
Soldier  performance  with  the  need  to  satisfy  equally  important,  practical  requirements. 

Suggestions  for  Future  Research 

There  are  several  avenues  for  future  research.  First,  future  research  should  consider  the 
effects  of  basic  and  technical  training  on  Soldier  proficiency  and  its  implications  for  Army 
classification.  That  is,  as  currently  computed,  estimates  of  aggregate  Soldier  performance,  such  as 
MPP,  do  not  consider  training  effects,  although  these  effects  are  present.  That  is,  while  test 
composites,  and  their  respective  weights,  are  corrected  to  the  Army  Input  population,  the  ASVAB- 
SQT  validities  on  which  they  are  based  are  conditioned  on  those  recruits  who:  (a)  successfully 
passed  training  (i.e.,  lower  performers  have  a  greater  probability  of  attriting  during  technical 
training);  and  (b)  experienced  the  performance-enhancing  effects  of  training  (i.e.,  as  training 
increases  recruit  performance).  Therefore,  the  contributions  of  basic  and  technical  training  to 
Soldier  proficiency  and  its  implications  for  classification,  and  measures  of  its  effectiveness  (i.e., 
MPP),  are  not  readily  understood.  More  importantly,  understanding  the  effects  of  training  on 
Soldier  proficiency  could  prove  beneficial  for  operational  reasons.  That  is,  when  making  entry- 
level  assignments,  considering  the  performance-enhancing  effects  of  training  is  likely  to  open  up  a 
greater  number  of  jobs  any  given  recruit  could  be  assigned  to,  particularly  those  recruits  whose  AA 
scores  substantially  limit  the  number  of  jobs  for  which  he/she  qualifies.  If  more  jobs  are  available 
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to  a  recruit,  the  differentiation  (differential  validity)  among  the  jobs  has  a  greater  opportunity  to 
contribute  to  classification  efficiency,  since  the  ability  of  job-specific  test  composites  to 
differentiate  among  the  jobs  is  maximized.  This  will  be  especially  true  for  recruits  with  lower  AA 
scores,  who  traditionally  have  been  difficult  to  place.  In  sum,  methodologies  that  integrate  the 
effects  of  training  and  classification  into  a  single  personnel  management  framework  are  likely  to 
prove  beneficial  for  both  research  and  operational  purposes. 

Second,  over  the  past  decade,  there  has  been  increasing  recognition  that  the  nature  of 
work  and  job  performance  in  many  jobs  has  changed  (Howard,  1995;  Ilgen  &  Pulakos,  1999; 
Schmitt  &  Chan,  1998).  In  some  cases,  these  changes  have  been  dramatic.  The  increasing  use 
of  technology,  the  shift  to  team-based  or  service-oriented  work  arrangements,  and  rapid 
environmental  and  organizational  changes  all  necessitate  different  types  of  skills  and  aptitudes 
(i.e.,  adaptability,  interpersonal  skills,  customer  service  orientation),  or  different  levels  of  the 
same  aptitudes,  for  successful  performance  than  did  jobs  a  decade  ago.  These  changes  have  not 
only  impacted  jobs  in  the  civilian  sector,  but  military  jobs  as  well.  For  example,  over  the  past 
dozen  years,  the  technological  complexity  of  jobs,  such  as  Combat  (CO),  has  grown  substantially 
with  the  advent  of  new  technologies  and  weaponry.  However,  the  current  analyses  are  based  on 
a  population  of  jobs  and  predictor-criterion  data  that  are  fifteen  years  old.  As  a  result,  the 
proposed  test  composites,  and  their  respective  weights,  are  unlikely  to  reflect  these  changes.  If 
the  population  of  Army  jobs  and  magnitude  of  predictor-criterion  relations  have  changed,  as  a 
function  of  systematic  changes  in  the  content  and  performance  requirements  of  these  jobs, 
classification  decisions  based  on  the  proposed  composites  are  likely  to  be  affected. 

Third,  future  research  is  needed  that  constructs  and  evaluates  the  classification  potential  of 
composites  that  include  noncognitive  variables,  such  as  personality  and  vocational  interests.  There 
is  a  growing  body  of  research  from  applied  psychology  showing  that  personality  and  other 
noncognitive  variables  are  predictive  of  performance  across  a  wide  range  of  occupations  and  are 
generally  uncorrelated  with  and  add  incremental  validity  over  and  above  cognitive  ability,  and  that 
predictive  validities  differ  by  job  content  and  performance  requirements  (Barrick  &  Mount,  1991; 
Barrick,  Mount,  &  Judge,  2001;  Hough  &  Fumham,  2003;  McHenry,  Toquam,  Hanson,  & 
Ashworth,  1990;  Mount  &  Barrick,  1995;  Mount,  Barrick,  &  Stewart,  1998;  Schmidt  &  Hunter, 
1998).  More  specific  to  the  Army,  research  from  Project  A  demonstrated  that  predictor  composites 
reflecting  different  mixes  of  cognitive  aptitudes,  personality  traits,  interests,  and  background 
characteristics  more  strongly  differentiated  and  predicted  performance,  including  technical 
proficiency,  across  jobs  than  predictor  composites  based  on  aptitude  alone  (Wise,  McHenry,  & 
Campbell,  1990).  On  the  basis  of  this  evidence,  noncognitive  variables,  specifically  personality 
and  vocational  interests,  could  greatly  extend  the  classification  potential  of  cognitively-based 
composites.  We  presently  know  of  few  studies  (e.g.,  Wise  et  al.,  1990)  that  consider  the 
effectiveness  of  composites  incorporating  noncognitive  variables  in  a  multiple  job  context. 


Conclusion 

In  an  earlier  report  (Diaz  et  al.,  2004)  we  independently  replicated  Zeidner,  Johnson,  and 
colleagues’  method  of  empirically  deriving  AA  composites,  including  the  9  AA  composites 
currently  in  operational  use.  The  primary  purpose  of  the  current  report  was  to  evaluate  the 
efficacy  of  the  proposed  AA  composites,  and  corresponding  job  families,  to  meet  the  Army’s 
classification  objectives.  Presently,  there  has  been  a  long-standing  debate  about  the  degree  to 
which  regression-weighted  composites  tailored  to  specific  jobs  based  on  test  batteries  assessing 
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specialized  aptitudes  and  abilities,  such  as  the  ASVAB,  produce  differences  in  validities  that 
represent  “true”  between-job  differences  and  not  differences  due  to  sampling  error  or  other 
artifacts  (Hunter,  1983, 1985;  Hunter  et  al.,  1985;  Schmidt  et  ah,  1988;  Schmidt  et  ah,  1981; 
Zeidner  &  Johnson,  1994;  Zeidner  et  ah,  1997).  To  address  these  issues,  we  tested  the  stability 
and  differential  validity  of  the  proposed  AA  composites  and  accompanying  job  families, 
particularly  the  17  and  150  relative  to  the  9  AAs,  and  their  practical  effects  on  classification 
efficiency,  as  measured  by  MPP. 

Overall,  our  findings  supported  the  continued  use  of  standardized  AA  composites  when 
assigning  recruits  to  entry-level  MOS  based  on  the  9  job  families  proposed  by  Zeidner  and 
colleagues.  We  recommended  these  composites  over  the  17  and  150  AA  composites  for  two 
reasons.  First,  consistent  with  recent  research  (Zeidner  et  ah,  2003b),  moving  from  9  to  17  AA 
composites  did  not  produce  practically  significant  increases  in  either  overall  MPP  or  MPP  by 
MOS.  Second,  and  more  importantly,  the  9  AA  composites  based  on  standardized  weights 
displayed  operationally  desirable  properties  relative  to  unstandardized  composites.  More 
specifically,  standardized  composites  can  be  expected  to  more  effectively  balance  the 
optimization  of  aggregate  soldier  performance  with  the  need  to  satisfy  quality  distribution 
requirements.  In  summary,  when  coupled  with  the  administrative  costs  and  other  management- 
related  issues  associated  with  changing  existing  cut  scores  based  on  the  existing  composites,  the 
technical  and/or  practical  advantages  to  adopting  the  17  or  150  AA  test  composites,  as  currently 
constructed,  are  few. 
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APPENDIX  A:  IDENTIFYING  WEAK  TEST  COMPOSITES  BY  JOB  FAMILY 


A-l 


Table  1 

Chi-Square  (%2)  Significance  Tests  and  Observed 


Significance  Values  (p)  of  Full  Test  Composites  for  9 
and  1 7  Job  Families _ _ 


Job  Family  Configuration 

9  17 

JF 

X 

JF 

X 

P 

CL 

11558.522 

.0005 

CL1 

6818.712 

.0005 

CO 

4799.789 

.0005 

CL2 

6060.484 

.0005 

EL 

8026.876 

.0005 

COl 

2005.642 

.0005 

FA 

3900.326 

.0005 

C02 

4918.017 

.0005 

GM 

7268.939 

.0005 

ELI 

5036.132 

.0005 

MM 

15148.526 

.0005 

EL2 

3429.786 

.0005 

OF 

6980.239 

.0005 

EL3 

1773.143 

.0005 

SC 

4153.739 

.0005 

FA 

3900.326 

.0005 

ST 

5224.271 

.0005 

GM1 

3952.679 

.0005 

GM2 

3411.606 

.0005 

MM1 

13237.128 

.0005 

MM2 

2610.497 

.0005 

OF 

6980.239 

.0005 

SC 

2831.936 

.0005 

ST1 

2047.810 

.0005 

ST2 

945.654 

.0005 

ST3 

2857.677 

.0005 

Table  2 

Chi-Square  (%2)  Significance  Tests  and  Observed  Significance  Values  (p)  of  Full 


Test  Composites  for  150  Job  Families 


JF 

— ■ — r4— 

X 

P 

JF 

X 

P 

JF 

x j 

P 

11B 

709.341 

.0005 

41C 

89.921 

.0005 

67U 

641.690 

.0005 

11C 

1327.798 

.0005 

44B 

618.032 

.0005 

67V 

293.000 

.0005 
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Table  3 

T-Tests  and  Observed  Significance  Values  (p)  of  Test  Composite  Weights  by  ASVAB  Subtestfor  9  Job  Families^ 
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Table  4 

T-Tests  and  Observed  Significance  Values  (p)  ofTest  Composite  Weights  by  ASVAB  Subtestfor  17  Job  Families 
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15E+16J _  -L514  0.0773  0J299  -L440  0.0802  0.1499 _ L042  0.0787  0.2973 _ 395 

Note.  Effect  sizes  that  are  not  statistically  significant  (p  >  .05)  are  bold  and  italicized. 
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Table  1 

Differences  in  Predicted^  Performance  Scores  by  Job  Family  (9  Job  Family  Configuration) 
Job  Family  CL  CO  EL  FA  GM  MM 
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Table  2 

Differences,  in  Predicted.  Performance^  Scores  fry  Job  Family  (1 7  Job  Family  Configuration) 

Job  Family  CL1  CL2  COl  CQ2  ELI  EL2  EL3 

CL1 


ON 

r- 

OO 

wo 

<N 

© 

wo 

vo 

CN 

© 

vo 

© 

r- 

VO 

i— t 

CO 

l> 

<N 

00 

© 

CO 

00 

00 

VO 

r- 

ON 

i 

CO 

© 

00 

r- 

ON 

VO 

1 

CO 

r- 

wo 

VO 

oo 

ON 

CN 

vo 

wo 

wo 

i—i 

!— 1 

VO 

wo 

1—* 

ON 

cd 

© 

i> 

<N 

© 

© 

d 

<N 

i 

VO 

wo 

vo 

r- 

VO 

1> 

ON 

vo 

ON 

CO 

wo 

i— < 

ON 

vo 

On 

y—< 

© 

l> 

wo 

ON 

VO 

© 

CO 

l> 

1 

ON 

CO 

CO 

1> 

ON 

r- 

r- 

1 

t-H 

© 

vo 

r- 

© 

wo 

CN 

^r 

l—H 

00 

ON 

oo 

vo 

ON 

VO 

(N 

<N 

<N 

1—1 

cd 

<N 

On 

© 

d 

© 

r- 

r- 

wo 

vo 

<N 

00 

wo 

<N 

CO 

WO 

oo 

i— < 

VO 

CO 

© 

CO 

oo 

00 

On 

OO 

ON 

vo 

r- 

ON 

VO 

r- 

wo 

ON 

ON 

OO 

CO 

1 

vo 

wo 

co 

oo 

vo 

vo 

vo 

OO 

<N 

r- 

wo 

. 

1 

ON 

r- 

On 

oo 

Tf 

vo 

CO 

co 

On 

wo 

/“N 

© 

p 

co 

wo 

cq 

r- 

tt 

CO 

oo 

On 

Ti¬ 

© 

o 

*— i 

© 

■— « 

© 

wo 

© 

© 

© 

© 

*-* 

ed 

x> 

Tf 

wo 

f-H 

vo 

t> 

tj- 

CO 

On 

© 

00 

vo 

*8 
•  ^ 

wo 

vo 

r- 

WO 

<N 

CO 

oo 

00 

wo 

wo 

<N 

Tf 

o 

© 

Tf 

ON 

VO 

CO 

CO 

ON 

''t 

OO 

^3h 

i 

(N 

00 

co 

CO 

r- 

i—i 

1— t 

Tj- 

wo 

r- 

vo 

i 

Tf 

wo 

00 

CO 

VO 

<N 

ON 

© 

r3" 

CO 

CO 

vo 

*^2 

<N 

Tf 

ON 

wo 

© 

ti- 

CM 

tq 

© 

vq 

r- 

3 

© 

© 

<© 

tt 

© 

© 

© 

<N 

© 

S 

oo 

VO 

On 

r- 

© 

wo 

wo 

r- 

Tt 

00 

oo 

1 

OO 

q 

© 

00 

vo 

ON 

Tf 

© 

VO 

© 

OO 

CO 

CO 

r- 

CO 

a> 

VO 

<N 

00 

vo 

On 

Tf 

<N 

<N 

1— ( 

i— H 

w^ 

00 

1 

Tfr 

vo 

<N 

vo 

vo 

Tf 

r- 

vo 

VO 

vo 

a 

1 

wo 

ON 

fH 

ON 

On 

ON 

r- 

1—1 

r- 

r- 

r- 

00 

1 

co 

<N 

rq 

(N 

<N 

CN 

(N 

wo 

CO 

oo 

© 

C> 

d> 

© 

(N 

© 

wo 

© 

© 

© 

(N 

a 

<D 

Tf 

ON 

wo 

VO 

© 

Tt 

CO 

ON 

vo 

vo 

00 

(N 

Tf 

d 

co 

r- 

CO 

r- 

CO 

wo 

Tf 

On 

vo 

(N 

wo 

wo 

CO 

W 

on 

vo 

CO 

co 

r- 

i-H 

CO 

00 

On 

00 

y—t 

i— < 

oo 

C/3 

1 

(N 

ON 

r- 

<N 

Tt 

ON 

co 

l> 

oo 

CO 

t-' 

2 

1 

Tf 

00 

Tf 

i> 

tj- 

CO 

co 

co 

wo 

1— 1 

00 

r- 

oo 

© 

wo 

© 

OO 

i> 

00 

1— < 

vq 

rq 

cq 

cq 

(N 

*— J 

Tf; 

co 

<N 

rt 

© 

r-’ 

ON 

wo’ 

<N 

oi 

vd 

o 

© 

(N 

VO 

© 

oo 

ON 

oo 

VO 

r- 

© 

© 

© 

© 

oo 

ca 

C/3 

o 

CN 

On 

r_< 

vo 

<N 

vo 

CO 

ro 

On 

On 

00 

i—i 

ri- 

vo 

oo 

OO 

Tf 

© 

VO 

© 

VO 

i> 

© 

© 

r> 

<N 

r- 

© 

d 

i> 

1 

r- 

© 

wo 

vo 

ON 

r- 

vo 

<N 

VO 

CO 

ON 

<N 

CN 

1 

00 

<N 

Tf 

On 

r- 

00 

<N 

CN 

1— 1 

1— ( 

<N 

vo 

i— ( 

© 

Ui 

p 

VO 

o 

CO 

vo 

1— < 

tt 

vq 

(N 

OO 

r-4 

wo 

oo 

On 

© 

<£ 

cd 

cs 

f-H 

© 

T“H 

Tf 

<N 

i— 1 

<N 

<N 

© 

cd 

cd 

1— H  : 

© 

i— • 

<u 

4-> 

\o 

fN 

VO 

on 

co 

wo 

wo 

oo 

ON 

Tf 

i—i 

i— t 

<N 

ON 

© 

Tl* 

’55 

co 

© 

r- 

CO 

t- 

CO 

CO 

r- 

© 

© 

CO 

CO 

© 

ON1 

o 

oo 

© 

© 

CO 

CN 

wo 

Tt 

r- 

oo 

CO 

vo 

r- 

wo 

CO 

OO 

& 

wo 

t- 

ON 

ON 

On 

ON 

<N 

r-H 

<N 

i— i 

© 

CO 

oo 

© 

VO 

co 

<N 

cn 

VO 

<N 

r- 

r- 

r- 

CO 

T|- 

<N 

CO 

CO 

© 

<N 

OO 

♦— i 

r\ 

t> 

© 

<N 

VO 

CO 

<N 

co 

vq 

<N 

<S 

wo 

'-o’ 

O 

00 

vd 

vd 

wo 

<N 

wo 

ON 

oo 

© 

r-* 

cd 

cd 

4-i 

(N 

O 

03 

o 

o 

d 

C3 

'C 

03 

> 

(Sj  i— i  CN 

JOO  J 

en 

J  J  < 

% 

(N 

s 

i 

<N 

s 

Uh 

U 

H 

(N 

H 

co 

H 

o  o  u  w 

W  W  PH 

O 

o 

s 

s 

O 

00 

00 

00 

00 

B-3 


Table  2  (cont’d) 

Differences  in  Predicted  Performance  Scores  by  Job  Family  (1 7  Job  Family  Configuration) 

Composite  GM1  GM2  MM1  MM2  OF _ SC _ ST1 _ ST2 _ ST3 


B-4 


Table  3 

Expected.  Reduction  in  Total.  R-Squared  (R2)  from  Combining  Job  Families j _ 

Job  Family  CL1  CL2  COl  CQ2  ELI _ EL2  EL3 
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Table  3  (cont’d) 

Expected  Reduction  in  Comjwsite  Validity  (R2)  from  Combining  Job  Families _ 

Composite _ GM1 _ GM2 _ MM1 _ MM2 _ OF _ SC _ ST1 
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APPENDIX  C:  SAS  PROGRAMS  FOR  RUNNING  K-FOLD  DOUBLE  CROSS- 
VALIDATION  DESIGN  FOR  ESTIMATING  MPP 
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Program  applying  the  MPP  K-Fold  Validation  macro  program  . . . 


options  formchar= ' | -++++++ '  nodate  nonumber; 

/*  ***  EDIT  AS  NEEDED  ***  Input  data  library  */ 

libname  kfvLib  "D:\NEW  AA\SAS  Workspace\KFVPGMDISTN\InputData"; 
/*  ***  Edit  AS  NEEDED  ***  Output  data  library  */  • 

libname  mppsim  "D:\NEW  AA\SAS  Workspace\KFVPGMDISTN\MmmSimData"; 


/*  Include  OPJM  K-Fold  Validation  macros  */ 

%let  PRINT=OFF; 

/*  ***  EDIT  AS  NEEDED  ***  Directory  where  programs  are  located  */ 
filename  KFVPGM  'D:\NEW  AA\SAS  Workspace\KFVPGMDISTN\Programs '  ; 
%include  KFVPGM (CreateSampleABC) ; 

%include  KFVPGM (ComputeDescriptives) ; 

% include  KFVPGM (St eplA_correction_unreliability_KFV) ; 

%include  KFVPGM (SteplB_correction_range-restriction_KFV) ; 

%include  KFVPGM (Step2_JF_Validities_KFV) ; 

% include  KFVPGM (Step3_Best_Positive_Weights_KFV) ; 

% include  KFVPGM(Step4&5_b_to_uk_values_KFV) ; 

% include  KFVPGM (ComputeCriterionScores) ; 

%include  KFVPGM (OptimalAssignmetPredictedScores ) ; 

%include  KFVPGM (KFoldEvalMpp); 


/*  ***  EDIT  AS  NEEDED  ***  Dump  log  output  to  file  ...  */ 

%let  RUNLOG=" D:\NEW  AA\SAS 

Workspace\KFVPGMDISTN\MmmSimData\RUN_KFoldEvalMpp_02022004 . TXT"  ; 
proc  printto  log=&RUNLOG;run; 


/*  ***  EDIT  AS  NEEDED  ***  PREFIX  of  output  files  */ 

%let  RUNPREFFNAME=YPP; 

/*  ***  EDIT  AS  NEEDED  ***  Random  seed. 

*/ 

/*  NOTE:  Use  SAME  SEED  for  all  simulation  configuration  to  do  paired 
comparisons  */ 

%let  RUNSEED=1001 ; 


/*  ***  Edit  RUNx  macro  vars  in  %LET  STATEMENTS  to  specify  simulation  problem 
parameters  */ 

%let  RUNREPSTART=1 ; 

%let  RUNREPEND=4 9 ; 

/**  JF=9  ,  UNEQUAL  Variance  **/ 

%let  RUNJF=JF9; 
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%let  RUNBETACONSTRAINT=NONE;  /*  P0SITIVE1=JZV,  P0SITIVE2=HumRR0,  NONE=LSE 
*/ 

%let  RUNTIER=TIER1 ;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUN  FNAME= &  RUN  PRE  FFNAME . _&RUNJF . _&  RUNBETACON S  TRAI NT . _&RUNTIER . _&RUNREPSTART . _& 
RUNREPEND; 

/* 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP (mppsim. &RUNFNAME,  &RUNSEED, &RUNJF, &RUNBETACONSTRAINT, &RUNTIER,  &R 
UNREPSTART, &RUNREPEND)  ; 

*/ 

%let  RUNJF=JF9; 

%let  RUNBETAC0NSTRAINT=P0SITIVE1 ;  /*  POSITIVEl=JZV,  POSITIVE2=HumRRO, 

NONE=LSE  */ 

%let  RUNT I ER=T I ER1 ;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUNFNAME=&RUNPREFFNAME . _&RUN JF .  _&RUNBETACONSTRAINT . _&RUNTIER . _&RUNREPSTART . _& 
RUNREPEND; 

proc  datasets  library=mppsiin;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP (mppsim. &RUNFNAME,  &RUNSEED, &RUNJF, &RUNBETACONSTRAINT, &RUNTIER, &R 
UNREPSTART, &RUNREPEND) ; 

%let  RUNJF=JF9; 

%let  RUNBETACONSTRAINT=POSITIVE2;  /+  POSITIVEl=JZV,  POSITIVE2=HumRRO, 
NONE=LSE  */ 

%let  RUNT I ER=T I ER1 ;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUNFNAME=&RUNPREFFNAME._&RUNJF._&RUNBETACONSTRAINT._&RUNTIER._&RUNREPSTART._& 

RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP (mppsim. &RUNFNAME,  &RUNSEED, &RUNJF, &RUNBETACONSTRAINT, &RUNTIER,  &R 
UNREPSTART, &RUNREPEND) ; 

/**  JF=9  ,  EQUAL  Variance  **/ 

%let  RUNJF=JF9; 

%let  RUNBETACONSTRAINT=NONE;  /*  P0SITIVE1-JZV,  POSITIVE2=HumRRO,  NONE=LSE 
*/ 

%let  RUNTIER=TIER2;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUNFNAME=&RUNPREFFNAME._&RUNJF._&RUNBETACONSTRAINT._&RUNTIER._&RUNREPSTART._& 

RUNREPEND; 

/* 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP (mppsim. &RUNFNAME,  &RUNSEED, &RUNJF,  &RUNBETACONSTRAINT, &RUNTIER, &R 
UNREPSTART, & RUNREPEND)  ; 

*/ 


%let  RUNJF=JF9; 

%let  RUNBETAC0NSTRAINT=P0SITIVE1 ;  /*  POSITIVEl=JZV,  POSITIVE2=HumRRO, 

NONE=LSE  */ 


\ 
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%let  RUNTIER=TIER2 ;  /*  TIERl=unequal  variance,  TIER2  equal 

variance  */ 

%  let 

RUNFNAME=&RUNPREFFNAME._&RUNJF._&RUNBETACONSTRAINT._&RUNTIER._&RUNREPSTART._& 

RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP  (mppsim.  &RUNFNAME,  &RUNSEED,  &RUNJF,  &RUNBETACONSTRAINT ,  &RUNTIER,  &R 
UNREPSTART,  & RUNREPEND)  ; 

%let  RUNJF=JF9 * 

%let  RUNBETACONSTRAINT=POSITIVE2 ;  /*  P0SITIVE1= JZV,  POSITIVE2=HumRRO, 

NONE=LSE  */  mxrtno  - 

%let  RUNTIER=TIER2 ;  /*  TIERl-unequal  variance,  TIER2  equal 

variance  */ 

%  1  e  t 

RUNFNAME=&RUNPREFFNAME  ._&RUNJF._&RUNBETACONSTRAINT  ._&RUNTIER._&RUNREPSTART 

RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run;  '  mT__  r_ 

IKFoldEvalMPP  (mppsim.  &RUNFNAME,  &RUNSEED,  &RUNJF,  &RUNBETACONSTRAINT,  &RUNTIER,  & 
UNREPSTART,  &RUNREPEND) ; 

/**  jf=17  ,  UNEQUAL  Variance  **/ 

%let  RUNBETACONSTRAINT=NONE;  /*  POSITIVEl=JZV,  P0SITIVE2=HumRR0,  NONE=LSE 

%let  RUNTIER=TIER1;  /*  TIERl-unequal  variance,  TIER2-equal 

variance  * / 

%  1  e  t 

RUNFNAME=&RUNPREFFNAME  . _&RUN JF .  _&RUNBETACONSTRAINT  .  _&RUNTIER .  _&RUNREPSTART  .  _& 
RUNREPEND; 

/* 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run;  „T1Mn,x-.D  rD 

%KFoldEvalMPP  (mppsim.  &RUNFNAME,  &RUNSEED,  &RUNJF ,  &RUNBETACONSTRAINT ,  &RUNTIER,  &R 
UNREPSTART,  & RUNREPEND) ; 

*/ 


%let  RUNJF=JF17;  ^ 

%let  RUNBETAC0NSTRAINT=P0SITIVE1;  /*  POSITIVEl=JZV,  POSITIVE2-HumRRO, 

NONE=LSE  */  .  „„„„„  , 

%let  RUNTIER-TIER1;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%  let 

RUNFNAME=&RUNPREFFNAME._&RUNJF._&RUNBETACONSTRAINT._&RUNTIER._&RUNREPSTART._& 

RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

% KFol dEva 1MPP (mppsim. &RUNFNAME, &RUNSEED, &RUNJF, &RUNBETACONSTRAINT, &RUNTIER,  &R 
UNREPSTART, &RUNREPEND) ; 

%let  RUNBETACONSTRAINT=POSITIVE2 ;  /*  P0SITIVE1=JZV,  P0SITIVE2=HumRR0, 

%let~RUNTIER=TIERl;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 
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%let 

RUN  FNAME= &  RUN  PRE  FFNAME .  RUN  JF .  RUNBET ACONS  TRAI  NT  .  _&  RUNT  I ER . 
RUNREPEND; 


&RUNREPSTART.  & 
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proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP (mppsim. &RUNFNAME, &RUNSEED,  &RUNJF, &RUNBETACONSTRAINT, &RUNTIER, &R 
UNREPSTART, &RUNREPEND) ; 

/**  JF=17  ,  EQUAL  Variance  **/ 

%let  RUNJF=JF17 ; 

%let  RUNBETACONSTRAINT=NONE;  /*  POSITIVEl=JZV,  POSITIVE2-HumRRO,  NONE-LSE 
*/ 

%let  RUNTIER=TIER2;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUNFNAME=&RUNPREFFNAME . _&RUN JF . _&RUNBETACONSTRAINT .  _&RUNTIER . _&RUNREPSTART . _& 
RUNREPEND; 

/* 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

IKFoldEvalMPP {mppsim. &RUNFNAME, &RUNSEED,  &RUNJF, &RUNBETACONSTRAINT, &RUNTIER, &R 
UNREPSTART, &RUNREPEND) ; 

*/ 


%let  RUNJF=JF17 ; 

%let  RUNBETAC0NSTRAINT=P0SITIVE1 ;  /*  P0SITIVE1=JZV,  P0SITIVE2=HumRR0, 

NONE=LSE  */  , 

%let  RUNTIER=TIER2 ;  /*  TIERl=unequal  variance,  TIER2— equal 

variance  */ 

%let 

RUNFNAME=&RUNPREFFNAME._&RUNJF. &RUNBETACONSTRAINT ,_&RUNTIER.__&RUNREPSTART 

RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP  (mppsim.  &RUNFNAME,  &RUNSEED,  &RUNJF,  &RUNBETACONSTRAINT,  &RUNTIER,  &R 
UNREPSTART, & RUNREPEND) ; 


%let  RUNJF-JF17 ; 

%let  RUNBETACONSTRAINT=POSITIVE2 ;  /*  POSITIVES JZV,  POSITIVE2=HumRRO, 

NONE=LSE  */  o  i 

%let  RUNTIER=TIER2;  /*  TIERl=unequal  variance,  TIER2=equal 

variance  */ 

%let 

RUN  FNAME = &  RUN  P  RE  F  FNAME  .  _&RUN JF .  _&RUNBETACONSTRAINT .  _&RUNTIER .  _&RUNREPSTART . 
RUNREPEND; 

proc  datasets  library=mppsim;  delete  &RUNFNAME;  quit;  run; 

%KFoldEvalMPP  (mppsim.  &  RUN  FNAME,  &RUNSEED,  &RUNJF,  &RUNBETACONSTRAINT ,  &RUNTIER,  &R 
UNREPSTART, &RUNREPEND) ; 


/*  RESET  PROC  PRINTO  */ 
proc  printto;run; 
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/*+***+*+*  +  +  ***+*******+ **************************************  *************** 
Program  that  partitions  the  data  into  samples  A(PP),  B (AA) ,  and  C (Cross) 

***  REQUIRED  IN  SIMULATION  *** 

****************************************************************************/ 

/*  Initialize  total  number  of  MOS  */ 
data  _null_; 

set  kfvLib. entrymosl55  nobs=numMOS; 
put  numMOS=numMOS ; 
call  symput ( ' NUMMOS ' , numMOS) ; 
stop; 
run; 


/*  read  MOS  sample  size  in  Army  input  data  */ 

Macro  ReadMOSsizeArray; 

array  asizeN{ &NUMMOS }  _temporary_; 
array  asizeC{ &NUMMOS}  _temporary_; 
do  i=l  to  & NUMMOS; 

set  kfvLib. entrymosl55  (keep  =  mosnumid  obsN  obsC) ; 
asizeN{mosnumid}  =  obsN; 
asizeC{mosnumid)  =  obsC; 
end; 

drop  obsC  obsN; 

%mend; 


/*  partition  the  data  into  3  parts:  sampleDataA  sampleDataB  sampleDataC  */ 
%macro  CreateSampleABC (IREP, REPSEED, NREPSAMPLEC) ; 
data  SamplePartition; 

%  Re  adMOS  s i ze Array ; 
do  mosnumid=l  to  &NUMMOS; 
sizeC  =  asizeC{mosnumid} ; 
id  =  0; 

do  i=l  to  asizeN {mosnumid} ; 
id+1; 

inC  =  (  (&IREP-1) *sizeC  <  i  <=  ( &IREP) *sizeC  ); 
if  (AinC)  then 

randToSort  =  1+uniform ( &REPSEED) ; 
else 

randToSort  =  -( 1+uniform ( &REPSEED) ) ; 
output; 
end; 
end; 

drop  sizeC  i  inC; 
run; 

proc  sort  data=SamplePartition; 

by  mosnumid  randToSort; 
run; 

data  SamplePartition; 

% ReadMOSsizeArray ; 
do  mosnumid=l  to  &NUMMOS; 

sizeOverallC+asizeCfmosnumid} ; 


C-7 


end; 

retain  sizeA  sizeC; 
do  while (Alast) ; 

set  SamplePartition  end=last; 
by  mosnumid; 

if  first .mosnumid  then  do; 
sizeC  =  asizeC{mosnumid} ; 

sizeA  -  round( (asizeN{mosnumid}-sizeC) /2) ; 

nA  =  0; 
end; 

if  (randToSort<0)  then  do; 

/*  next  two  line  cycle  from:  1, 2, 3, 4 , &NREPSAMPLEC  */ 

irepc  =  mod (irepc, &NREPSAMPLEC) ; 

irepc+1; 

samplelD  -  'CM  left (put (irepc, 2. ) ) ; 
end; 

else  if  (  nA  <  sizeA)  then  do; 
samplelD  =  *Af; 
nA+1; 
end; 
else 

samplelD  =  1 B* ; 
output; 
end; 

keep  mosnumid  id  samplelD; 
run; 

proc  sort  data=SamplePartition; 

by  mosnumid  id; 
run; 

data  sampleDataA  sampleDataB 

%do  ISAMPLEC=1  % to  &NREPSAMPLEC; 

sampleDataC&ISAMPLEC 

%end; 

merge  kfvLib. kfvArmylnput  SamplePartition; 
by  mosnumid  id; 
if  (sampleID=l A* )  then 
output  sampleDataA; 
else  if  (samplelD^’B* )  then 
output  sampleDataB; 

%do  ISAMPLEC=1  %to  &NREPSAMPLEC ; 

else  if  ( sampleID=,’C&ISAMPLEC” )  then 
output  sampleDataC&ISAMPLEC; 

%end; 

run; 

%mend; 
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y***************************************************************************^ 

Compute  descript ives,  correlations,  and  covariances  for  samples  A(PP),  B(AA) 
***  REQUIRED  IN  SIMULATION  *** 


*****+************+**********★**********************************************/ 

%macro  ComputeDescriptives  ( SAMPLEDATA,  MEANDATA,  CORRDATA,  COVDATA,  COVDATA_ALL)  ; 

proc  means  da ta=& SAMPLEDATA  noprint; 
var  GS  AR  NO  CS  AS  MK  MC  El  VE  SQT; 
by  mosnumid; 
output  out=tmp; 
run; 

proc  transpose  data=tmp 

out = &MEAN DATA  (keep=  mosnumid  _name__  mean  std 
rename=  (_name^names  std=sd)  )  ; 
var  GS  AR  NO  CS  AS  MK  MC  El  VE  SQT; 
id  _stat_; 
by  mosnumid; 
run; 

proc  corr  da ta=& SAMPLEDATA  noprint 

outp=& CORRDATA  (where=  (_type_j= 1  CORR*  ) 
rename^  (_name_=names)  )  ; 
var  GS  AR  NO  CS  AS  MK  MC  El  VE  SQT; 
by  mosnumid; 
run; 

proc  corr  data=& SAMPLEDATA  noprint  cov 

outp=& COVDATA  ( wher e= (_t  ype_j= 1 COV 1 ) 
rename^ (_name_=names) ) ; 
var  GS  AR  NO  CS  AS  MK  MC  El  VE  SQT; 
by  mosnumid; 
run; 

proc  corr  da t  a= &  SAMPLEDATA  noprint  cov 

outp=&COVDATA_ALL  {where= (_type_= 1 COV1 ) 
rename^ (_name_=names) ) ; 
var  GS  AR  NO  CS  AS  MK  MC  El  VE  SQT; 
run; 


%mend; 
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Correcting  ASVAB-SQT  validity  coefficients  and  covariances 
for  criterion  unreliability 

OUTPUT  DATASETS:  Ryx_CorrectRelib,  Cyx_CorrectRelib_MOS 
INPUT  DATASETS:  Descript ive__MOS,  R_Samp_MOS,  Info_MOS 

***  REQUIRED  IN  SIMULATION  *** 

VC***************************************************************************/ 


%macro  CorrectUnreliability (Ryx_CorrectRelib_MOS, Cyx_CorrectRelib_MOS, 

Descript ive^MOS, R_Samp_MOS,  Info__MOS) ; 

/*  variable  names  for  ASVAB  subtests  */ 

%let  TESTNAME S-GS  AR  NO  CS  AS  MK  MC  El  VE; 

proc  iml; 

TestNames  =  { &TESTNAMES } ; 

/*  Read  ASVAB-SQT  correlations  and  numeric  ID  into  XYcorr  and  MOSNumID  */ 
use  &R__Samp_MOS; 

read  all  var  (TestNames )  where  (NAMES-” SQT” )  into  XYcorr; 
read  all  var  {MOSNUMID}  where  (NAMES-”SQT” )  into  MOSNumID; 
close  &  R_S  amp__MOS  ; 

/*  Read  SQT  reliabilities  into  vector  YYscal*/ 
use  &Info_MOS; 

read  all  var{RelYY}  into  YYscal; 
close  &Info__MOS; 

NumMOS  -  nrow (MosNumID) ;  /*  =nrow (CorMat )  */ 

NumTest  -ncol (TestNames) ; 

/*  correcting  validity  coefficients  for  attenuation  using  standard  formula 

*/ 

correctedRxy  =  XYcorr# ( 1/SQRT (YYscal) ) ; 

/*  initialize  corrected  covariance  matrix  */ 
correct edCVxy  -  repeat (0, NumMOS, NumTest ) ; 

/*  computing  corrected  covariances  —  one  ASVAB  subtest  column  at  a  time  */ 
use  &Descriptive_MOS; 

read  all  var{SD}  where (names-" SQT” )  into  sdY; 
do  iTest  -  1  to  NumTest; 
xName  -  TestNames [iTest ] ; 

read  all  var{SD}  where (NAMES— xName)  into  sdX; 
correctedCVxy [ , iTest]  =  correctedRxy [, iTest ] #sdY#sdX; 
end; 

close  &DescriptiveJMOS; 

/*  creating  dataset  of  MOS  corrected  SQT-ASVAB  correlations  */ 
create  &Ryx_CorrectRelib_MOS  var {MOSNUMID  &TESTNAMES}; 
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correctedRxy  =  MosNumID  | |  correctedRxy; 
append  from  correctedRxy; 
close  &Ryx_CorrectRelib_MOS ; 

/*  creating  dataset  of  MOS  corrected  SQT-ASVAB  covariances 
create  &Cyx  CorrectRelib_MOS  var{MOSNUMID  &TESTNAMES}; 
correctedCVxy  =  MosNumID  | I  correctedCVxy; 
append  from  correctedCVxy; 
close  &Cyx_CorrectRelib_MOS ; 

quit  ; 
run; 


%mend; 


Correcting  ASVAB-SQT  validity  coefficients  and  covariances  for 
range  restriciton. 

OUTPUT  DATASETS:  Ryx_CorrectRelibRange_MOS 

INPUT  DATASETS:  C_Ref Pop, C_Samp_MOS, Cyx_CorrectRelib_MOS 
***  REQUIRED  IN  SIMULATION  *** 


%macro  CorrectRangeRestriction (Ryx_CorrectRelibRange_MOS, 

C_RefPop, C_Samp_MOS, Cyx_CorrectRelib_MOS) ; 

/*  variable  names  for  ASVAB  subtests  */ 

%let  TESTNAMES=GS  AR  NO  CS  AS  MK  MC  El  VE; 

proc  iml ; 

TestNames  =  { &TESTNAMES } ; 

/*  read  REFERENCE  POPULATION  ASVAB'  subtests  covariance  */ 

use  &C  Ref Pop;  ,  ,  ^  _  _ 

read  all  var { &TESTNAMES }  where (_TYPE_=' COV'  &  names?TestNames)  into  PopCxx; 

close  &C_RefPop; 

SDx  =  sqrt (vecdiag ( PopCxx) ) ; 

/*  open  MOS  SQT-ASVAB  covariance  corrected  for  unreliability  */ 

use  &Cyx_CorrectRelib_MOS; 

read  all  var {MOSNUMID}  into  MosNumID; 

NumMOS  =  nrow (MosNumID) ; 

/*  open  MOS  SQT-ASVAB  sample  variance-covariance  —  no  correction  */ 
use  &C_Samp_MOS ; 

/*  create  output  data  for  range— restriction  corrected  validities  / 
create  &Ryx_CorrectRelibRange_MOS  var {MOSNUMID  & TESTNAMES} ; 

/♦looping  through  MOSs  listed  under  MOSTextID*/ 
do  idxMOS=l  to  NumMOS; 

/*  read  reliability  corrected  ASVAB-SQT  covariance,  uncorrected  ASVAB 

variance-covariance,  and  uncorrected  SQT  variance  from  iTH  MOS  */ 

setin  &Cyx  CorrectRelib_MOS; 

read  all  var { &TESTNAMES }  where (MOSNUMID=idxMOS)  into  Cxc;  *correctedCVxy; 

setin  &C  Samp_MOS;  ,  , 

read  all  var { &TESTNAMES }  where (names?TestNames  &  MOSNUMID-idxMOS)  into 

Cxx  * 

'read  all  var{SQT}  where (names=* SQT •  &  MOSNUMID=idxMOS)  into  Cyy; 

/*  compute  range-restriction  corrected  ASVAB-SQT  covariances  for  iTH  MOS 

*/ 

PopCxc  =  PopCxx*inv (Cxx) *Cxc ' ; 

PopCcc  =  Cyy+Cxc*inv (Cxx) * (PopCxc-Cxc' ) ; 
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/*  compute  range-restriction  corrected  ASVAB-SQT  correlations  for  iTH 
MOS*/ 

PopRxc  =  (l/SDx/*Sxvec*/) # (PopCxc)  # (1/sqrt (PopCcc) ) ; 

/*  append  iTH  MOS  SQT-ASVAT  correlations  to  output  data  */ 

TmpOutput  =  idxMOS  ||  PopRxc'; 
set  out  &Ryx_CorrectRelibRangeJVIOS; 
append  from  TmpOutput; 

end; 

close  &Cyx_CorrectRelib_MOS; 
close  &C_Samp_MOS; 

close  &Ryx_CorrectRelibRange_MOS; 

quit; 

run; 

%mend; 
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/**★********************************** 

Aggregating  corrected  ASVAB-SQT  validity  coefficients  by  job  family. 

***  REQUIRED  IN  SIMULATION  *** 

***+*+**********************************************************************/ 


JFValid  ( JF  VALIDITY_DATA,  MOS_VALI D I T Y_DAT A ,  JF_SOLUTION,  J F_CON F I G_DAT A )  ; 

/*  variable  names  for  ASVAB  subtests  */ 

%let  TESTNAMES=GS  AR  NO  CS  AS  MK  MC  El  VE; 

proc  iml; 

/*  open  data  containing  Job  Family  MOS  configuration  */ 
use  &  J  F_CON  F I G_DAT A ; 

read  all  var { &JF_SOLUTION}  into  JFSolVec; 

/*  Total  number  of  JF  in  JFSolVec  vector  */ 

NumJF  =  max (JFSolVec) ; 

/*  open  data  containing  reference  population  MOS  validities  */ 
use  &  MO  S_VAL I D I T Y_D AT A ; 

/*  create  output  data  set  for  aggregated  JF  validities  */ 
create  &JF_VALIDITYJDATA  var { &JF_SOLUTION  &TESTNAMES} ; 
set out  & JF_VAL I D I T Y_DAT A ; 

do  idxJF  =  1  to  NumJF; 

/*  locate  MOS  in  iTH  job  family  and  read  acquisition  weights  */ 
setin  &JF_CONFIG_DATA; 

MOSJFIDX  =  loc ( JFSolVec=idxJF) ; 

read  point (MOSJFIDX)  var(AcqN)  into  N_Wgt; 

/*  read  corrected  validities  of  MOSs  in  iTH  job  family  */ 
setin  &MOS_VALIDITY_DATA; 

read  point (MOS JFIDX)  var { &TESTNAMES }  into  XYvec; 

/*  aggregate  validity  coefficients  across  MOS  weighted  by  N  */ 

/*  -  note  job  family  index  is  concatenated  to  output  validity  vector  */ 
JFCorr  =  idxJF  ||  (diag(N_Wgt) *XYvec) [+, ] /sum(N_Wgt) ; 

append  from  JFCorr; 

end; 

close  &JF_CONFI G_D AT A ; 
close  & MOS_VAL I D I T Y_DAT A ; 
close  & JF  VALIDITY_DATA; 


quit ; 

run; 

%mend; 
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Computing  Beta  Weights  by  job  family. 

*  Use  macro  argument  CONSTRAINT  to  obtain  different  solutions: 

NONE  =  no  constraint  on  subtest  weights 
P0SITIVE1  =  Postive  weights  using  Zeidner- Johnson-Vladimirsky  stopping 
rule 

P0SITIVE2  =  Postive  weights  —  ignoring  solutions  with  negative  weights 
***  REQUIRED  IN  SIMULATION  *** 

•A-******-*-****-*’-*--*:-*--*-************************-********'*********  ******************/ 

%macro  BetaWeights  (BETADATA,  COVDATA,  VALIDITYDATA,  JFSOLUTION,  CONSTRAINT)  ; 

/*  variable  names  for  ASVAB  subtests,  excluding  NO  and  CS  */ 

%let  TESTNAMES=GS  AR  AS  MK  MC  El  VE; 

%let  CORRDATA=TMPCORRDATA; 

proc  iml; 

TestNames  =  { &TESTNAMES  SQT } ; 

NTests  =  ncol (TestNames) ; 

_TYPE_=  {  "MEAN" ,  "STD”,  "N"}//j  (NTests,  1,  "CORR")  ; 

_NAME_=j (3, 1, "" ) lit  (TestNames) ; 

call  symput ( 1 MNTESTS 1 , char (NTests )) ; 

/*  Used  later  for  CONSTRAINT=POSITIVE  +/ 
call  symput  ( 1  MNTESTS__ASVAB 1 ,  char  (NTests-1 )  )  ; 

do  i=l  to  NTests; 
if ( i < 1 0 )  then 

MTESTNAME  =  concat ( fMTESTNAME 1 , char (i,  1 , 0 ) ) ; 

else 

MTESTNAME  -  concat ( 1 MTESTNAME char (i,  2 , 0 )) ; 
call  symput (MTESTNAME, TestNames [i]  )  ; 
end; 

use  &COVDATA; 

read  all  var { &TESTNAMES }  where ( (Names?TestNames )  &  (Names A?”SQT") )  into 
RXX; 

close  &COVDATA; 

SXX_INV  -  sqrt (diag ( 1/RXX) ) ; 

RXX  -  SXX_INV*RXX*SXX_INV; 

XMEAN  =  j (1, NTests, 0) ; 

XSTD  =  j (1, NTests, 1) ; 

/*  NOT  actual  sample  sizes,  but  does  not  matter  for  estimation  */ 

XN  =  j (1, NTests, 10000) ; 

/*  Read  Validity  Data  Matrix  —  Note  that  MOS/JF<->Row  */ 
use  &VALIDITYDATA; 

read  all  var { &TESTNAMES }  into  RXY_ALL ; 
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read  all  var { & JFSOLUTION }  into  JFNO_ALL; 
close  &VALIDITYDATA; 

/*  For  each  job  family,  read  validities  and  create  correlation  matrix  / 

create  &CORRDATA (Type=corr)  var  {{  & JFSOLUTION  _TYPE _ NAME_} || TestNames ) ; 

do  iJF  =  1  to  nrow ( JFNO_ALL) ; 

IdxJF  =  JFNO_ALL [iJF] ; 

& JFSOLUTION  =  j (nrow (_TYPE_) , 1, IdxJF) ; 

RXY  =  RXY_ALL [ i JF , ] ; 

XCORR  =  (RXX//RXY) I | (t (RXY) //l) ; 

XDATA  =  XMEAN//XSTD//XN//XCORR; 

%do  i=l  %  to  &&MNTESTS; 

&&MTESTNAME&i  =  XDATA [ , &i] ; 

%end; 


append; 

end;  /*  ENDOF:  do  iJF  =  1  to  nrow ( JFNO_ALL)  */ 


close  &CORRDATA; 

quit; 

run; 


%if  &CONSTRAINT=NONE  %then  Ido; 

%let  MODELOPTION=NOINT ; 

%let  KEEPOTHER=; 
lend; 

%else  %do; 

%let  MODELOPTION=NOINT  SELECTION=RSQUARE  B; 

%let  KEEPOTHER=_RSQ _ P_; 

lend; 


proc  reg  data=&CORRDATA 

out  e  s  t = &  BET ADAT A  ( keep=& JFSOLUTION  &TESTNAMES  &KEEPOTHER) 
NOPRINT; 

model  SQT  =  &TESTNAMES  /  &MODELOPTION; 
by  & JFSOLUTION; 
quit; 
run; 


Zeidner-Johnson-Vladimirsky  Non-negative  Beta  Weights^APproach^^^^ 


%if  &C0NSTRAINT=P0SITIVE1  %then  %do; 
proc  sort  data=&BETADATA; 

•  by  & JFSOLUTION  descending  _P_  descending  _RSQ_; 
run; 


/*  Oragnize  all  possible  solutions  using  two  data  sets:  J 
/*  TmpBetaPositive:  solutions  with  non-negative  weights  ^  / 
/*  TmpBetaMax:  solutions  with  maximum  R  for  each  JF  &  no.  subtests  pair  */ 
data  TmpBetaPositive 
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TmpBetaMax 

( keep=& JFSOLUTION  __RSQ _ P_)  ; 

array  Beta  { &MNTESTS_ASVAB}  &TESTNAMES; 
set  &BETADATA; 

by  &JFSOLUTION  descending  _P_  descending  _RSQ_; 

NegativeWgtFlag  =  0; 
do  i=l  to  &MNTESTS_ASVAB; 

if  (Beta{i}=. )  then  Beta{i}  -  0; 

NegativeWgtFlag  =  NegativeWgtFlag  or  (Beta{i}<0); 
end; 

/*  output  all  solutions  without  negative  weights  */ 
if  ( ANegativeWgtFlag)  then 
output  TmpBetaPositive; 

/*  output  subset  with  maximum  R  overall  for  given  number  of  subtests  */ 
if  (First. _P_  and  A First . & JFSOLUTION)  then  do; 

_P_  = 

output  TmpBetaMax; 
end; 

run; 

/*  output  positive  weighted  solutions  with  R2  >=  max  R2  in  the  next  level 

*/ 

data  TmpCompare; 

keep  &  JFSOLUTION  &TESTNAMES  _RSQ_; 

merge  TmpBetaPositive  TmpBetaMax  (rename=  (_JRSQ_=Rmax)  )  ; 
by  &  JFSOLUTION  descending  _P_; 
if(_RSQ_  >=  Rmax)  then  output; 
run; 

/*  output  only  weights  with  maximum  number  of  subsets  for  job  family  */ 
data  &BETADATA; 
set  TmpCompare; 

by  & JFSOLUTION  descending  _RSQ_; 
if  First. & JFSOLUTION  then  output; 
run; 

%end; 

/****************************************************************** 

HumRRO  Simple  Non-negative  Beta  Weights  Approach: 

-  Entirely  ignore  solutions  with  negative  weights. 
******************************************************************/ 

%if  &C0NSTRAINT=P0SITIVE2  %then  %do; 

/*  keep  only  solutions  with  all  positive  weights  */ 
data  TmpBetaPositive; 

array  Beta  { &MNTESTS_ASVAB)  &TESTNAMES ; 

set  &BETADATA; 

do  i=l  to  &MNTESTS__ASVAB ; 

if  (Beta{ i }=. )  then  Beta{i}  =  0; 
else  if  (Beta{i}<0)  then  delete; 
end; 
run; 

proc  sort  data=TmpBetaPositive; 

by  & JFSOLUTION  descending  _RSQ_; 
run; 
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/*  output  all  positive  weights  solution  with  maximum  R2  */ 
data  &BETADATA; 

keep  & JFSOLUTION  &TESTNAMES  _R SQ_; 
set  TmpBetaPositive; 
by  & JFSOLUTION  descending  _RSQ_; 
if  First .& JFSOLUTION  then  output; 
run; 

%end; 

%mend; 
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Computing  Beta  Weights  by  job  family. 
***  REQUIRED  IN  SIMULATION  *** 


%macro  UKWeights  (UK  DAT  A,  CO  V  DAT  A,  POPDATA,  BETADATA,  JFNUM,  TYPE)  ; 

/*  variable  names  for  ASVAB  subtests,  excluding  NO  and  CS  */ 

%let  TESTNAMES=GS  AR  AS  MK  MC  El  VE; 

proc  iml; 

Subtest  =  { &TESTNAMES } ; 

/*  predictor  correlation  matrix  for  Army  Input  Population*/ 
use  &COVDATA; 

read  all  var { &TESTNAMES)  where (names?Subtest )  into  CovMat; 
close  &COVDATA; 

use  &  POPDATA; 

read  all  var{SD}  where (test ?Subtest)  into  SDvec; 

%if  &TYPE=TIER1  %then  %do; 

read  all  var {MEAN}  where (test? Subtest)  into  Means; 

%end; 

close  & POP DATA; 

SDProd  =  1/ (SDvec#SDvec) ; 

R  =  CovMattSDProd' ; 

/*reading  in  all  JFs  into  JFNO_ALL*/ 
use  & BE TAD AT A; 

read  all  var{&JFNUM}  into  JFNO_ALL; 

NumJF  =  nrow ( JFNO_ALL) ; 

/^creating  SAS  dataset  containing  u  weights  and  k  values  for  all  JFs  */ 
create  &UKDATA  var{JFNO  &TESTNAMES  k}; 

do  idxJF=l  to  NumJF; 

/^converting  beta  weights  to  b-weights  for  MOS-level*/ 
setin  &BETADATA; 

read  all  var { &TESTNAMES }  where ( & JFNUM=idxJF)  into  ObsBeta; 
bweights  =  ObsBeta# ( 1/SDvec ') ; 

/^transform  b-weights  to  u  and  k  values  for  Tier2  */ 

%if  &TYPE=TIER2  %then  %do; 

/*  composite  multiplier*/ 

CM  =  20/ (10* (SQRT (bweights*R*bweights ' ) ) ) ; 

/*  calculate  U  and  K  values  */ 

Uvec  =  diag (CM) *bweights ; 

K  =  (SUM (Uvec) *50)-100; 
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%end; 


/^transform  b-weights  to  u  and  k  values  for  Tierl  */ 

%else  %if  &TYPE-TIER1  %then  %do; 

/*  calculate  U  and  K  values  */ 

Uvec  -  bweights; 

/*  sum  ASVAB  means  weighted  by  their  respective  b-weight*/ 
K  =  SUM (Uvec#Means  ' )  ; 

%end; 

/^merging  u  values  with  k  value  and  adding  a  column  for  JFNO*/ 
UKvec  =  J (nrow (K) , 1 , idxJF)  II  Uvec  M  K; 

append  from  UKvec; 

end; 

close  & UK DAT A; 
close  &BETADATA; 

quit; 

run; 

%mend; 
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/**********************w* *************************************************** 

Compute  criterion  scores  for  individuals  in  sample  &CrossSample 

Note:  TYPE=ASSIGN  prepare  cost  data  in  preparation  for  optimal  classification 
using  PROC  TRANS. 

***  REQUIRED  IN  SIMULATION  *** 

******************•*******************+**************************************/ 


%macro 

ComputeCriterionScores (CriterionData, CrossSample,  JFWgtData, JFSOLN, TYPE) ; 

%if  &TYPE=ASSIGN  %then  %do; 

proc  sort  data=kfvlib. entryMOS150  out=tmp; 

by  & JFSOLN; 
run; 

proc  means  data=tmp  noprint; 
var  allocKFV; 
by  & JFSOLN; 

output  out=AllocData  sum=; 
run; 

%end; 

proc  iml; 

use  &JFWgtData; 

read  all  var{K  GS  AR  AS  MK  MC  El  VE}  into  WgtMat; 
close  & JFWgtData; 

use  &CrossSample; 

read  all  var  {GS  AR  AS  MK  MC  El  VE}  into  X; 
close  &CrossSample; 

X  -  repeat (-l,nrow(X)  ,  1)  I  I  X; 

Y . =  X*t (WgtMat); 

create  &CriterionData  var ( " JF1" : "& JFSOLN" ) ; 

%if  &TYPE=ASSIGN  %then  %do; 
use  AllocData; 

read  all  var {allocKFV}  into  tmpvec; 
tmpvec  =  t (tmpvec); 
append  from  tmpvec; 
close  AllocData; 

%end; 

append  from  Y; 
close  &CriterionData; 

quit; 

run; 

%mend; 
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Optimally  assign  persons  to  JFSOLN  jobs.  The  program  then  compute  their 
predicted  performance  in  their  respective  optimal  jobs. 

***  REQUIRED  IN  SIMULATION  *** 

****************************************************************************/ 


%macro  OptimalAssignmentPredictedScores (OPJMData, EvalData, AssignData, JFSOLN) ; 

data  TmpAssign; 
set  &AssignData; 
if  _n_>l  then  do; 

person  =  'P'  ||  put (_n_-l, z4 . ) ; 

supply  =  1; 
end; 
run; 

proc  trans  cost=TmpAssign  out=AssignSolution  maximum; 
tailnode  person; 
headnode  JF1-&JFS0LN; 
supply  supply; 
run; 


data  tmp; 

array  AJF150{150}  JF1-JF150;  /*  not  all  150  use  all  the  time  */ 
set  AssignSolution  end=last; 
if  _n_>l  and  Alast; 

do  & JFS0LN=1  to  150  until  (AJF150 {& JFSOLN } >0) ; end; 
randToSort  =  uniform(8000+&IREP) ; 
keep  person  & JFSOLN  randToSort; 
run; 

proc  sort  data=tmp; 

by  & JFSOLN  randToSort; 
run; 

proc  sort  data=kfvlib.entryMOS150  out=tmp2; 

by  & JFSOLN; 

'  run; 

data  tmp2; 
set  tmp2; 

do  i=l  to  allocKFV; 

output; 

end; 

keep  & JFSOLN  MOS150; 
run; 

data  MOS150assignIdx; 
merge  tmp  tmp2; 
by  & JFSOLN; 

keep  MOS150  & JFSOLN  person; 
run; 

proc  sort  data=MOS150assignIdx; 

by  person; 
run; 

proc  iml; 
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use  &EvalData; 

read  all  var ( " JF1 " : " JF150" )  into  matY; 
close  &EvalData; 

use  MOSISOassignIdx; 
read  all  var{MOS150}  into  MOS150idx; 
read  all  var{&JFSOLN}  into  JFSOLNidx; 
close  MOSISOassignIdx; 

assignN  =  nrow{matY); 
optY  =  repeat (0, assignN, 1) ; 
do  i=l  to  assignN; 

optY[i]  =  matY [i, MOSISOidx [i] ] ; 
end; 

outMat  = 

repeat (&IREP, assignN, 1) | | repeat {&IREPC, assignN, 1) | |MOS150idx| |JFSOLNidx| loptY 

create  &OPJMData  var{REP  REPC  MOS150  &JFSOLN  YPP}; 
append  from  outMat; 
close  &OPJMData; 

quit; 

run; 

%mend; 
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Program  implenting  the  K- Fold  validation  of  JZ  procedure 


%macro  KFoldEvalMPP (KFVOPJMDATA, RANDSEEDBASE,  JFSOLN,  BCONSTRAINT,  UTIER, 
REPSTART, REPEND)  ; 

/*  Currently  number  of  sample  C  partitions  is  fixed  */ 

%let  NUMREPC=5; 

%do  IREP=&REPSTART  %to  & RE PEND; 

%let  REPSEED=%eval { &RANDSEEDBASE+&IREP) ; 

/*  Partition  data  in  samples  A+B+C1+C2+ . . . +C&NUMREPC  */ 

%CreateSampleABC ( &IREP, &REPSEED, &NUMREPC) ; 

/*  Using  sample  A,  compute  weights  for  evaluation  of  MPP  —  always  based  on 
JF150  */ 

*  Compute  descriptive  stats; 

%ComputeDescriptives (SampleDataA, descripA, corrA, covA, covRefpopA) , 

*  Step  1  -  correct  MOS  validity  for  criterion  unreliability  and 
restriction  in  range; 

%Correct Unreliability (corXYA, corCVA, descripA, corrA,  kfvLib.entrymosl55) ; 

*  note:  range  restriction  correction  relative  to  Army  -  covRefpopA 
computed  from  input  data; 

%CorrectRangeRestriction (validityA, covRefpopA, covA, corCVA) ; 

*  Step  2  -  compute  JF  validities  —  corrected  to  Army  input  population; 
%JFValid(JF150validityA, validityA, JF150,  kfvLib.entrymosl55) ; 

*  Step  3  -  compute  beta  weights  —  CONSTRAINT=NONE  for  evaluation  sample 

A  * 

%BetaWeights ( JF150BetaA, covRefpopA, JFISOvalidityA,  JF150, NONE) ; 

*  Step  4&5  --  final  JF150  evaluation  weights  for  computing  MPP  —  note 

param  TIER1  ;  > 

%UKWeights ( JFISOukA, covRefpopA, kfvLib. PopDescripArmy, JF150BetaA, JF150, TIER1) ; 

/★  Using  sample  B,  compute  assignment  weights  based  on  solution  &JFSOLN  */ 

*  Compute  descriptive  stats; 

%ComputeDescriptives (SampleDataB, descripB, corrB, covB, covRefpopB) ; 

*  Step  1  -  correct  MOS  validity  for  criterion  unreliability  and 
restriction  in  range; 

%CorrectUnreliability (corXYB, corCVB, descripB, corrB, kfvLib . entrymosl55 ) ; 

*  note:  range  restriction  correction  relative  to  Army  -  covRefpopA 
computed  from  input  data; 

%CorrectRangeRestriction ( validityB, covRefpopB, covB, corCVB) ; 

*  Step  2  -  compute  & JFSOLN  JF  validities  —  corrected  to  Army  input 
population; 

% JFValid ( JFSOLNvalidityB, validityB, & JFSOLN, kfvLib . entrymosl55)  ; 

*  Step  3  -  compute  & JFSOLN  beta  weights  —  CONSTRAINT=NONE  for  evaluation 

sample  A  ; 
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%BetaWeights ( JFSOLNBetaB, covRefpopB, JFSOLNvalidityB,  &JFSOLN,  &BCONSTRAINT) ; 

*  Step  4  &5  —  final  &JFSOLN  assignment  weights  —  note  param  NOT  TIER2 
(NOT  want  visible  tier)  ; 

%UKWeights  ( JFSOLNukB,  covRefpopB,  kfvLib .  PopDescripArmy,  JFSOLNBetaB,  &JFSOLN,  &UT 
IER)  ; 

/*  Using  sample  C  subsets  */ 

%do  IREPC=1  %to  &NUMREPC; 

*  Compute  prediction/eval  criterion  scores  using  sampleA  MOS150  LSE 
wgts  (JFISOukA)  ; 

%Comput eCr it erionS cores (CrossDataEval/ Sample DataC& I REPC,  JFISOukA, JF150 , EVAL) ; 

*  Compute  analysis/assignment  criterion  scores  using  sampleB  JF 
solution  wgt  ; 

%ComputeCriterionScores (CrossDataAssign, SampleDataC&IREPC,  JFSOLNukB, &JFSOLN, A 
SSIGN) ; 

*  Run  optimal  assignment; 

%  Opt  imalAssignment  Predict  edScores  (RepOPJMdata,  CrossDataEval,  CrossDataAssign,  & 
JFSOLN) ; 

*  Append  OP JM  data  ; 

proc  append  base=&KFVOPJMDATA  data=RepOPJMdata; 
run ; 

%end; 

%end; 

%mend; 

/* 

%let  TMPJF= JF9; 

proc  datasets;  delete  TmpMpp&TMPJF;run; 

%KFoldEvalMPP (TmpMpp&TMPJF, 1001, &TMPJF , POSITIVE1, TIER2,  1,  1) ; 
proc  means  data=TmpMpp&TMPJF; 

var  YPP; 

*  class  REPC; 
run; 

%let  TMPJF= JF17 ;  . 

proc  datasets;  delete  TmpMpp&TMPJF; run; 

%KFoldEvalMPP (TmpMpp&TMPJF,  4  9, 1001, &TMPJF , POSITIVE1,  TIER2,  1,1) ; 
proc  means  data=TmpMpp&TMPJF; 

var  YPP; 
run; 

%let  TMPJF=JF150 ; 

proc  datasets;  delete  TmpMpp&TMPJF; run; 

%KFoldEvalMPP  (TmpMpp&TMPJF,  4 9,  1001,  &TMPJF ,  POSITIVE1 ,  TIER2 ,1,1)  ; 
proc  means  data=TmpMpp&TMPJF; 

var  YPP; 
run; 

*/ 
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Calculate  overall  MPP  by  replication  across  conditions: 

NUM_JF,  BETA_CONSTRAINT,  TIER(a.k.a.  EQUAL  VARIANCE) 

*************************★*************+************************************/ 


options  formchar=' I -++++++'  nodate  nonumber; 

/*  *+*  EDIT  AS  NEEDED  ***  Output  data  library  */ 

libname  mppsim  "D:\NEW  AA\SAS  Workspace\KFVPGMDISTN\MmmSimData"; 

data  ypp/view=ypp; 
set 

mppsim. ypp_j f 9_none_tierl_l_4  9  ( in=in_j  f 9_none_tierl ) 

mppsim . ypp_j f 9_none_t ier2_l_4  9  ( in=in_j  f 9_none_tier2 ) 

mppsim . ypp_j f 9_positivel_tierl_l_4  9  ( in=in_j f 9_positivel_tierl ) 
mppsim. ypp_j f 9_positivel_tier2_l_4 9  (in=in_j f 9_positivel_tier2 ) 
mppsim. ypp_j f 9_positive2_tierl_l_4 9  (in=in_jf9_positive2_tierl) 
mppsim. ypp  j f 9_positive2_tier2_l_49  (in=in_j f 9_positive2_tier2 ) 
mppsim. ypp_j  f 17_none_tierl_l_4  9  (in=in_j  f 17_none_tierl ) 

mppsim. ypp_jfl7_none_tier2_l_4 9  (in=in_jfl7_none_tier2) 

mppsim.  ypp_j  f  17_positivel_tierl_l_4  9  ( in=in_j  f  17_positivel_tier  1 ) 
mppsim. ypp~j  f 17_positivel_tier2_l_49 (in=in_jf 17_positivel_tier2 ) 
mppsim. ypp_j  f 17_positive2_tierl_l_4 9 { in=in_j  f 17_positive2_tier 1 ) 
mppsim. ypp  j f 17_positive2_tier2_l_4 9 ( in=in_j f 17_positive2_tier2 ) 

f 

if  in_jf 9_none_tierl  then  do; 

num_jf=9;  beta_constraint=0;  tier=l;  end; 
else  if  in_jf 9_none_tier2  then  do; 

num_jf=9;  beta_constraint=0;  tier=2;  end; 
else  if  in_jf 9_positivel_tierl  then  do; 

num_ j f =9 ;  beta_constraint=l;  tier=l;  end; 
else  if  in_j f 9_positivel_tier2  then  do; 

num_jf=9;  beta_constraint=l;  tier=2;  end; 
else  if  in_j f 9_positive2_tierl  then  do; 

num_jf=9;  beta_constraint=2;  tier=l;  end; 
else  if  in_j f 9_positive2_tier2  then  do; 

num_jf=9;  beta_constraint=2;  tier=2;  end; 
else  if  in_j  f 17_none_tierl  then  do; 

num_j f=17;  beta_constraint=0;  tier=l;  end; 
else  if  in_j f 17_none_tier2  then  do; 

num_j f=17;  beta_constraint=0;  tier=2;  end; 
else  if  in_j  f 17_positivel_tierl  then  do; 

num_jf=17;  beta_constraint=l ;  tier=l;  end; 
else  if  in_j f 17_positivel_tier2  then  do; 

num  jf=17;  beta_constraint=l ;  tier=2;  end; 
else  If  in_jf 17_positive2_tierl  then  do; 

num  j  f=17 ;  beta_constraint=2 ;  tier=l;  end; 
else  If  in_jf 17_positive2_tier2  then  do; 

num_j  f=17 ;  beta_constraint=2 ;  tier=2;  end; 

run; 

proc  means  data=ypp  noprint; 

by  num_jf  beta_constraint  tier  rep  repc; 
var  ypp; 
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output  out=mpp_overall;  run; 
data  mppsim.inpp_overall; 
set  mpp_overall; 
if  _stat__= 1  MEAN ' ; 

drop  _type _ freq _ stat_; 

rename  ypp=mpp; 
run; 
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APPENDIX  D:  PLOTTING  OF  MEAN  MPP  AND  STANDARD  ERROR  BY  MOS 
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Figure  3 

MPP  and  Standard  Error  by  MOS  and  Job  Family  Configuration  for  OF,  SC,  and  ST 
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Figure  4 

MPP  and  Standard  Error  by  MOS  and  Job  Family  Configuration  for  CL,  CO,  and  EL 
Job  Families  Using  Standardized  Test  Composites 
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APPENDIX  E:  MEAN  MPP  AND  STANDARD  ERROR  BY  MOS 
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Table  1 

Mean  MPP  and  Standard  Error  by  MOS  and  Job  Family  Configuration  Using 
Unstandardized  Test  Composites 


'i 

9  ' 

'  :Y1 

150 

*y\  ; 

Mean 

Std 

Mean 

Std 

Mean 

Std 

i  ' 

-0.3297 

0.0393 

-0.4058 

0.0403 

-0.4144 

0.0396 

2 

-0.4746 

0.0688 

-0.5741 

0.0622 

-0.0269 

0.1436 

:3;:; 

-0.4510 

0.0704 

-0.5479 

0.0625 

0.0433 

0.1425 

-0.3512 

0.1313 

-0.4513 

0.1081 

-0.5072 

0.1188 

-0.4456 

0.0619 

-0.0021 

0.1082 

0.0068 

0.1390 

I-- 

-0.4704 

0.1344 

0.0051 

0.1925 

0.2718 

0.2074 

-0.4556 

0.2633 

-0.0463 

0.2350 

0.1825 

_ 

0.2876 

8 

-0.1445 

0.1186 

-0.1415 

0.1017 

-0.2375 

0.0813 

9 

-0.1831 

0.4007 

-0.1281 

0.3687 

0.4469 

0.3883 

10 

-0.1736 

0.2025 

-0.1855 

0.1883 

0.7865 

0.1258 

M):-; 

-0.2128 

0.1520 

-0.2063 

0.1379 

0.3150 

0.2186 

12 

0.0123 

0.1389 

0.0391 

0.1454 

...... 

-0.1940 

0.2735 

0.0380 

0.1800 

0.0741 

0.1919 

0.0113 

0.2650 

14 

-0.1195 

0.2889 

-0.1167 

0.2930 

-0.3314 

0.3660 

15 

0.3002 

0.3634 

0.3326 

0.3944 

0.4838 

0.3600 

i!6 

0.0825 

0.2264 

0.1410 

0.2078 

0.0078 

0.3993 

17 

0.1651 

0.1445 

0.2012 

0.1439 

0.4694 

0.2095 
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0.1306 

0.1181 

0.1647 

0.1301 

0.2463 

0.1577 

|9;i 

0.0549 

0.0919 

0.0840 
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-0.0011 

0.1873 

20  ' 

-0.4915 

0.0533 

0.0054 

0.0969 

0.1133 

0.1190 
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-0.4824 

0.2015 

0.0704  ! 

0.2204 

0.1417 

0.2352 

22 

-0.5818 

0.3035 

0.0465 

0.3412 

0.4829 

0.2026 

23 

0.3981 

0.2120 

0.2349 

0.2400 

-0.2504 

0.3626 

M  1 

0.3878 

0.5078 

0.4778 

0.4459 

0.5613 

0.5241 

25 

0.3067 

0.2012 

0.1795 

0.2376 

-0.2318 

0.2239 

26 

0.4595 

0.4002 

0.2685 

0.3460 

0.1748 

0.3650 

27 

S.X.-":  • 

0.0484 

0.2164 

0.0522 

0.1981 

0.6086 

0.1467 

28 

0.4725 

0.1368 

0.6098 

0.1506 

0.5491 

0.1615 

29 

0.4311 

0.1285 

0.2465 

0.1536 

-0.0128 

0.1737 

30 

0.5357 

0.2962 

0.7513 

0.2586 

0.8172 

0.2517 

31 

0.3710 

0.2233 

0.4850 

0.2141 

-0.3702 

0.3323 

32 

0.4698 
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Cut  Score  Equating  Procedure 


Proposed  MOS-specific  cut  scores  using  alternative  composites  based  on  the  17  and  150 
job  family  configuration  were  computed  to  be  “equivalent”  to  existing  cut  scores  under  the 
existing  LSE  composites  based  on  the  9  job  family.  By  “equivalent”  we  mean  that  accessions  for 
a  given  MOS  based  on  the  alternative  composite  and  proposed  cut  score  for  the  MOS  will  have 
the  same  mean  predicted  performance  (MPP)  as  accessions  based  on  the  current  composite  and 
existing  cut  score  for  the  same  MOS. 

Generally,  varying  criterion  validities  can  be  expected  between  the  existing  and 
alternative  composites.  When  the  alternative  composite  has  higher  validity  compared  to  the 
existing  composite,  a  cut  score  that  is  lower  than  the  existing  value  can  produce  the  same  level  of 
MPP.  Thus,  the  benefit  of  improved  composite  validity  is  achieved  through  large  pool  of  eligible 
applicants  (i.e.,  higher  selection  ratio).  When  the  alternative  composite  has  lower  validity,  we 
can  compensate  for  its  lower  predictive  power  by  setting  a  higher  cut  score  that  will  produce  the 
same  level  of  MPP.  This  is  achieved  at  the  expense  of  a  smaller  eligible  pool  of  higher  average 
quality  applicants  than  the  existing  pool.  These  ideas  underlie  the  cut  score  equating  procedure 
describe  below. 


Denote  the  current  composite  for  the  mth  MOS  by  X 0 ,  with  validity  R0  and  cut  score 
currently  set  at  C0 ,  and  the  alternative  composite  by  Xn ,  with  validity  R„ .  Also  denote  the 
common  mean  and  standard  deviation  under  the  current  and  alternative  composites  (using  9, 17 
or  150  job  families)  by  constants  //,  =  100  and  a x  =  20 .  Using  these  notations  the  MPP  under 
the  current  and  alternative  composites  and  cut  scores  can  be  written  as 


MPPa 

MPP„ 


=  Rax 

■fe 

* 

V 

1 

"te 

l 

=  Rn  X 

'  Mx>c„  ~  Mx 

The  terms  inside  the  parentheses  are  simply  the  truncated  mean  of  the  composite  above  the 
respective  cut  scores  in  standardized  form. 

Under  the  equivalent  cut  score  condition,  we  have  MPPn  equals  MPP , ,  where  the  latter 
is  a  known  value  and  the  former  involves  the  unknown  cut  score  Cn  of  interest.  For  this  equality 
to  be  useful  in  solving  for  the  equivalent  alternative  cut  score,  first  we  need  to  expand  the 
truncated  means  of  the  composites  X0  and  X n  in  the  MPP  expressions  above  as  function  of  the 
composite  mean  and  variance.  Assuming  a  normal  distribution  for  the  composite  in  the  applicant 
population  and  using  standard  mathematical  results  (e.g.,  Greene  1997),  we  obtain 
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The  “hazard”  function  X  is  given  by  the  expression  below  in  terms  of  the  standard  normal  density 
<j>{z)  and  cumulative  distribution  <b(z). 


A(z)  = 


<&) 

1  -  0(z) 


Setting  MPP  under  current  and  alternative  composites  and  cut  scores  equal,  and  after 
some  algebraic  manipulations  to  separate  known  and  unknown  terms  on  either  side  of  the 
equation,  we  obtain 


(V 

i 

UJ 

K  °  x  j 

=  1 


C„  ~MX 


Thus,  to  obtain  the  unknown  equivalent  cut  score  Cn,  we  solve  for  the  zero  of  the  non-linear 
equation 


0  =  11 


C„  ~  Mx 


-L 


where  the  constant  L  is  simply  the  known  value  of  the  entire  expression  on  the  left-hand  side  of 
the  preceding  equality  relationship.  The  solution  to  this  problem  was  carried  out  using  the 
Newton  iterative  method.  The  SAS  program  implementation  of  the  entire  cutscore  equating 
procedure  appears  on  the  next  page. 


» 
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Data  for  Input  to  Cut  Score  Equating  Problem 


MOS-JF-CutScore  Configuration: 

(1)  mos  =  MOS  ID 

(2)  mosnumid  =  numeric  ID  ( 1, 2,  .  .  • ,  155)  _  ,  TT71  cn 

(3)  jf0103__9  j f 0103_17  jf0103_150  =  numerice  ID  for  JF9,  JF17,  and  JF150  as 

of  JAN 03 

(4)  c0  -  cut  score  based  on  JF9  ocmposite 

Input  Data  to  Cut  Score  Equating  Problem: 

(1)  zcO  =  standardized  cut  score  on  JF9 

(2)  qO  =  selection  ration  based  on  cO  and  JF9  composite  (l-probnorra(zcO) ) 

(3)  rO  =  validity  of  JF9  composite 

(4)  ppO  =  predicted  performance  based  on  cO  and  JF9  composite  of  MOS 

(5)  rl  =  validity  of  JF17  composite 
Output  Data  from  Cut  Score  Equating  Problem. 

(6)  zcl  =  standardized  cut  score  on  JF17  composite 

(7)  ql  =  selection  ratio  on  JF17  composite  and  cut  score 

(8)  cl  =  cut  score  on  JF17  composite  that  gives  the  same  predicted 

?**f******!*****************************************************************/ 


Cut  Score  Equating  Calculations 

NOTE:  MOS  in  JAN 03  file  that  not  found  in  the  SQT  data  are  excluded 

ASVAB  validities 

for  these  MOS  are  not  available. 


The 


%let  dataMosCsConfig=lcs .mosCurCS; 

%let  dataRXY155=lvalid. YouthValid; 

%let  dataRXY9=lvalid. JF9YouthValid; 

% macro  CutScoreEquate (dataRXY17,  dataYouthCov, dataCSsoln,  AltNumJF) ; 
%let  PI=3. 14159265; 
proc  iml; 


Newton  procedure  for  solving  root  of  LAMBDA  function^^^^^^ 


start  F  LAMBDA(z);  .  , 

phiz  =  exp (0.5* (-z##2) ) / sqrt ( 2*&PI ) ;  lambda  =  phiz/ ( 1-probnorm 

return (lambda)  ; 
finish  F_LAMBDA; 
start  DF  LAMBDA (z) ; 

phiz  =  exp (0.5* ( —  z# # 2 ) ) / sqrt (2*&PI) ;  dphiz  =  -z*phiz;  pnormz  - 
probnorm(z); 

dlambda  =  (  dphiz* ( 1-pnormz)  +  phiz##2  )  /  (1  pnormz) ##2; 

return (dlambda) ; 
finish  DF_LAMBDA; 

start  SOLVE_LAMBDA(lambdaConstant, initZ, epsZ,  maxlter, opt) ; 

if  (opt[l])  then 

print  lambdaConstant [label="lambdaConstant" 
format=10.8] , initZ [label="initZ"  format=10.8] ; 


(z 


)  ; 
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zO-initZ;  minDif f-epsZ;  zDiff=l;  maxiter=20; 
do  iter=l  to  maxiter  while  (  zDif  f  >ininDif  f )  ; 

zl  =  zO  -  (F_LAMBDA(zO) -lambdaConstant) /DF_LAMBDA(zO) ; 
zDiff  =  abs(zl-zO); 
zO  =  zl; 

y  if  (opt  [2])  then 

print  iter  [label=MiterH  format=3.0]  zO [label="ZSOLN"  format=10.8] 
zDiff [ label="zDif f "  format=10 . 8] ; 

end; 

*  return ( zO ) ; 
finish  SOLVE_LAMBDA ; 

/*  Testing  Code 

zcSOLN  -  1.987654; 

lambdaLHS  -  F_LAMBDA ( zcSOLN) ; 

zcSolved  =  S OLVE_LAMB DA (lambda LHS, 0, le-6, 20) ; 
print  zcSolved [label=" zcSolved"  format=10 . 8 ] ; 

*/ 

Z**************************************************************************** 

Read  MOS- JF-CutScore  Configuration  Data 
****************************************************************************/ 

use  &dataMosCsConf ig; 

read  all  var {mosnumid}  into  mosNumlD; 
read  all  var { j f 0103_9 }  into  JF9; 
read  all  var {cutscore}  into  JF9CS; 
read  all  var { j f 0103_&AltNumJF}  into  JFalt; 
close  SdataMosCsConf ig; 

*print  mosNumlD  JF9  JFalt  JF9CS; 

z* *************************************************************************** 

Read  MOS155,  JF9,  JFalt  Youth  ASVAB  validities,  and  Youth  ASVAB  covariance 

matrix 

*********************+******************************************★***********/ 
TestNames  =  {GS  AR  AS  MK  MC  El  VE}; 
use  &dataRXY155; 

read  all  var (TestNames)  into  rxyl55; 
close  &dataRXY15.5; 
use  &dataRXY9; 

read  all  var (TestNames)  into  rxy9; 
close  &dataRXY9; 
use  &dataRXY17; 

read  all  var (TestNames )  into  rxyl7; 
close  &dataRXY17; 
use  &dataYouthCov; 

read  all  var  (TestNames )  where (names ?TestNames )  into  cxxYouth; 

*  close  &dataYouthCov; 

Sprint  rxyl55  rxy9  rxyl7  rxxYouth; 
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**************************************************************************** 
Quantities  for  Current  JF9  composite  that  are  available  or  readily  computed: 


(1)  zcO  =  standardized  cut  score  on  JF9 

(2)  qO  =  selection  ration  based  on  cO  and  JF9  composite  (l-probnorm(zcO) ) 

(3)  rO  =  validity  of  JF9  composite 

(4)  ppO  =  predicted  performance  based  on  cO  and  JF9  composite  of  MOS 


Quantities  for  "New"  JFalt  composite  that  are  available  or  readily  computed: 
Known  quantities  from  "new"  JFalt  Composite: 

(5)  rl  =  validity  of  JFalt  composite 

Quantities  for  "New"  JFalt  composite  that  are  caculated  through  cut  score 
equating: 

(6)  zcl  =  standardized  cut  score  on  JFalt  composite 

(7)  ql  =  selection  ratio  on  JFalt  composite  and  cut  score 

(8)  cl  =  cut  score  on  JFalt  composite  that  gives  the  same  predicted 
performance 


meanComp=100;  stdComp=20; 
stdXX  =  sqrt (vecdiag (cxxYouth) ) ; 

rxxYouth  =  diag (1/stdXX) *cxxYouth*diag (1/ stdXX) ; 
numMos  =  nrow (mosNumID) ; 

zcO=repeat (0, numMos) ;  q0=repeat (0, numMos ) ;  r0=repeat (0, numMos) , 
ppO=repeat (0, numMos) ; 

zcl=repeat ( 0 , numMos ) ;  ql=repeat ( 0, numMos ) ;  rl=repeat (0/ numMos) / 
JFaltCS=repeat (0, numMos) ; 


do  idxMos=l  to  numMos; 


zcO [idxMos]  =  ( JF9CS [idxMos] -meanComp) / stdComp; 
qO [ idxMos ]  =  1  -  probnorm ( zcO [ idxMos] ) ; 

rO [idxMos]  =  sqrt (rxyl 55 [idxMos, ] *inv (rxxYouth) *t (rxy9 [JF9 [idxMos] , ] ) ) ; 
ppO [idxMos]  =  rO [idxMos] * (meanComp  +  stdComp* F_LAMBDA ( zcO [idxMos] )) ; 


rl [idxMos]  = 

sqrt (rxyl55 [idxMos, ] *inv (rxxYouth) *t (rxyl7 [JFalt [idxMos ] , ] ) ) ; 
lambdaLHS  =  (rO [idxMos] /rl [idxMos] ) *F_LAMBDA ( zcO [idxMos] ) ; 
zcl [idxMos]  =  SOLVE_LAMBDA ( lambdaLHS, zcO [idxMos ] , le-6,  20,  { 1  1}); 
ql [idxMos]  =  1  -  probnorm ( zcl [idxMos] ) ; 

JFaltCS [idxMos]  =  meanComp  +  stdComp* zcl [idxMos] ; 


end; 

matOut  =  mosNumID  | I  JF9CS  | |  zcO  | I  qO  ||  rO  | |  rl  II  zcl  II  ql  II 
JFaltCS; 

create  &dataCSsoln  var [mosNumID  JF9CS  zcO  qO  rO  rl  zcl  ql  JF&AltNumJF . CS } ; 
append  from  matOut; 
close  &dataCSsoln; 


quit; 

run; 

%mend; 

options  mprint^l; 

%CutScoreEquate ( 1 valid . JF17 YouthValid, 1 valid . PopCovYouth, lcs . csSolnFixPP__JF17 
17 )  ; 

%CutScoreEquate (lvalid. Jf 150youthvalid, 1 valid. PopCovYouth, lcs . csSolnFixPP_JFl 

50,150) ; 


V 


A 


F-6 


Table  1 

New  and  Old  Cut  Scores  and  Qualification  Rales  by  MOSJor  1 7  Job  Families 


MOS 

JF9CS 

ZC0I 

Q0 

R0 

R1 

ZC1 

Q1 

JF17CS 

1 

87 

-0.65 

0.742153889 

0.528374924 

0.50900736 

-0.615296654 

0.730820587] 

87.69406692" 

2 

87 

-0765; 

0.742153889 

0.602825356 

0.580085098  ‘ 

-0.614256995 

0.730477242 

87.71486011 

3 

87 

-0.65 

0.742153889 

0.590782909 

0.568950082 

-0.615003971" 

0.730723951 

87.69992059 

'  4 

87 

-0.65 

0.742153889 

0.540846301 1 

0.5205612681 

-0.614468113 

0.730546981 

87.71063775 

5 

87 

-0.65, 

0.742153889 

0.593138316 

0.630243661 

-0.705168449 

0. 75964729o 

85.89663103 

6 

87 

-0.65 

0.742153889 

0.59534116 

0.632041602 

-0.704398351 

0.759407634 

85.91203298 

8 

91 

-0.45 

0.67364478 

0.624690167 

0.624690167 

-0.45 

0.67364478 

91 

9 

91 

-0.451 

0.67364478 

0.56965701 1 

0.569657011 

-0.45 

0.673644781 

91 

To 

91 

-0.45 

0.67364478 

0.66029159 

0.66029159 

-0.45 

0.67364478 

91 

11 

91 

-0.45 

0.67364478 

0.662381536 

0.662381536 

-0.45 

0.67364478 

91 

12 

95 

-0.25 

0.598706326 

0.663324105 

0.663324105 

-0.25 1 

0.598706326 

95 

14 

98 

-0.11 

0.539827837 

0.648680415 

0.648680415 

-0.1 

0.539827837 

98 

22 

87" 

-0.65 

0.742153889 

0.605869684 

0.643843124 

-0.705268283 

0.759678353 

85.89463434 

24 

87 

-0.65 

0.742153889 

0.620224189 

0.66069355 

-0.707436999 

0.7603525241 

85.85126002 

26 

92 

-0.4 

0.655421742 

0.689912396 

0.704544844 

-0.421 706029' 

0.663380199 

91.56587941 

29 

102 

0.1 

0.460172163 

0.5921857 

0.590830448 

0.103006391 

0.458978949 

102.0601278 

33 

98 

-0.1 

0.539827837 

0.669540892! 

0.669540892 

-0.1 

0.539827837 

98 

35 

89 

-0.55 

0.708840313 

0.610008827 

0.6045850711 

-0.541342954 

0.705864391 

89.17314091 

37 

107 

0.35 

0.363169349 

0.701824913 

0.70111151 

0.351488998 

0.362610762 

107.02978 

39 

98 

-0.1 

0.539827837 

0.62863297 

0.62863297 

-oTi 

0.539827837 

98 

40 

116 

0.8 

0.211855399 

0.665774129 

0.663984154 

0.8047492041 

0.21048221 

116.0949841 

!  42 

102 

0.1 

0.460172163 

0.667514629 

0.678873927 

0.077982631 

0.468920935 

101.5596526 

43 

107 

0.35 

0.363169349 

0.668178174 

0.678647341 

0.327356329 

0.371 6991 961 

106.5471266 

44 

102 

0.1 

0.460172163 

0.685673394 

0.694533268 

0.083228333 

0.466834992 

101.6645667 

45 

"98 

-0.1 

0.539827837 

0.647524571 

0.657463354 

-0.118158605 

0.547029006 

97.6368279 

48 

88 

-0.6 

0.725746882 

0.72296745 

0.756598437 

-0.642488143 

0.739721857 

87.15023714 

49 

97 

-0.15 

0.559617692 

0.735289835 

0.769205125 

-0.202108333 

0.580083983 

95.95783334 

50 

88 

-0.6 

0.725746882 

!  0.600596715 

0.628873893 

-0.642986236 

0.739883484 

87.14027529 

51 

93 

-0.35 

0.636830651 

0.640425362 

0.672029756 

-0.400670342 

0.655668575 

91.98659317" 

52 

“97 

-0.15 

0.559617692 

0.694022416 

0.690916029 

-0.144746289 

0.557544405 

97.10507422 

53 

97 

h-0.15 

0.559617692” 

0.71979739 

0.754128421 

-0.203820078 

0.580752949 

95.92359843” 

56 

93 

-0.35 

0.636830651 

0.735260276 

0.770509576 

-0.399274616 

0.655154566 

92.01450767 

58 

88 

-0.6 

0.725746882 

0.701389192 

0.670573296 

-0.557137068 

0.711283108 

88.85725864 

59 

88 

-0.6 

0.725746882 

0.686933742 

0.657246038 

-0.557860406 

0.711530145 

88.84279188 

60 

88 

-0.6 

0.725746882 

0.656507343 

0.627799471 

-0.557346082 

0.711354502 

88.85307835 

61 

93 

-0.35 

0.636830651 

0.728627437 

0.721701283 

-0.33980615 

0.632998742 

93.20387701 

62 

102 

0.1 

0.460172163 

0.60610754 

0.621789864 

0.06675345 

0.473388991 

101.335069 

63 

97 

-0.15 

0.559617692 

0.680025098 

0.712658695 

-0.204139466 

0.580877743 

95.91721068 

64 

97 

-0.15 

0.559617692 

0.709182712 

0.743268433 

-0.204220618 

0.58090945 

95.91558764 

65 

92 

-0.4 

0.655421742 

0.746809578 

0.76818708 

-0.42913755 

0.666088437 

91.41724899 

66 

. 96 

-0.2 

0.579259709 

0.664528937 

0.646312705 

-0.167998996 

0.566707966 

96.64002007 

67 

"  104 

0.2 

0.420740291 

0.724644707 

0.69259301 

0.262882774 

!  0.396320467 

105.2576555 

70 

87 

-0.65 

0.742153889 

0.70000591 

1  0.693833377 

-0.641822058 

0.739505637 

87.16355884 

71 

88 

-0.6 

0.725746882 

0.740144864 

0.707759246 

-0.557318261 

0.711344999 

88.85363479 

Note.  MOS  =  MOS  ID  Number;  JF9CS  =  Old  AA  Cut  Score  (9  JF  Level);  ZCO  =  Standardized  Old  AA 
Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  =  Validity  of  9  JF-Level  AA  Composite;  R1 
=  Validity  of  17  JF  Level  AA  Composite;  ZC1  =  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF17CS  =  New  AA  Cut  Score  (17  JF  Level). 


Table  1  (cont’d) 


IMOSIJF9CS!  ZCO]  QO  i 

R0  !  R1 

ZC1  1  Q1  ! 

JF17CS 

72 

88 

-0.6 

0.725746882 

0.705386736 

0.674175432 

-0.556822855 

0.711175766 

88.86354291 

73 

88 

-0.6 

0.725746882 

0.641538019 

0.614321678 

-0.558659727 

0.711803015 

88.82680545 

74 

.87 

-0.65 

0.742153889 

0.713812452 

0.713335988 

-0.649384606 

0.741955094 

87.01230788 

75 

97 

-0.15 

0.5596176921 

0.764372563 

0.764445456 

-0.15011154 

0.559661692 

96.9977692 

76 

“97 

-0.15 

0.559617692 

0.792072668 

0.791226469 

-0.148749321 

0.559124279 

97.02501359 

77 

97 

-0.15 

0.559617692 

0.750551996 

0.75097059^ 

-0.150652098 

0.559874919 

96.98695804 

78 

87 

-0.65 

0.742153889 

0.676698726 

0.678247461 

-0.652105523 

0.742833449 

86.95788953 

79 

87 

-0.65 

0.742153889 

0.64366095 

0.645477037 

-0.652594721 

0.742991205 

86.94810558 

”81 

97 

-0.15 

0.559617692 

0.728881979 

0.7264861 851 

-0.146145427 

0.558096707 

97.07709145 

82 

102 

0.1 

0.460172163 

0.69485919 

0.693992438 

0.101637274 

0.459522296 

102.0327455 

!  83 
84 

87 

97 

-0.65, 

0.7421 538891 

0.750439786 

0.749837689 

-0.649260225 

0.741914905 

87.01479551 

-0.1 5  j 

0.559617692 

0.691086211 

0.690103098 

-0.148334177 

0.558960478 

97.03331646 

|  85 

102 

0.1 

0.460172163 

0.701452311 

0.729104488 

0.049870208 

0.480112909 

100.9974042 

86 

”97 

-0.15 

0.559617692 

0.70697104 

0.703505772 

-0.14424477 

0.557346405 

97.1151046 

87 

102 

0.1 

0.460172163 

0.717587198 

0.707256229 

0.119096867 

0.452599306 

102.3819373 

"  88 

'102 

0.1 

0.460172163 

0.627235496 

0.62447777 

0.105785404 

0.457876309 

102.1157081 

89 

102 

0.1 

0.4601 721631 

0.711505261 

0.71170251 

0.099636559 

0.46031 64341 

101.9927312 

90 

102 

0.1 

0.460172163 

0.672302894 

0.672714302 

0.099197971 

0.460490543 

101.9839594 

91 

102 

0.1 

0.460172163 

0.738440547 

0.743938095 

0.09029515 

0.464026337 

101.805903 

92 

102 

0.1 

0.460172163 

0.784300107 

0.782027342 

0.10380861 

0.458660617 

102.0761722 

93 

102 

0.1 

0.460172163 

0.784078298 

0.786851176 

0.095375689 

0.462008213 

101.9075138 

94' 

102 

0.1 

0.460172163 

0.71409059 

0.729673218 

0.071872091 

0.47135185 

101.4374418 

95 

93 

-0.35 

0.636830651 

0.721649898 

0.739391879 

-0.375701582 

0.646430619 

92.48596837 

97 

93 

-035 

0.636830651 

0.637569126 

0.65821653 

-0.383662211 

0.649385588 

92.32675579 

99 

107 

0.35 

0.363169349 

0.423410229 

0.411633457 

0.391656386 

0.34765606 

107.8331277 

101 

92 

-0.4 

0.655421742 

0.698706419 

0.699100407 

-0.400585981 

0.655637516 

91 .98828038 

105 

92 

'  -0.4 

0.655421742 

0.65003742 

0.650371295 

-0.400533773 

0.655618293 

91 .98932455 

106 

103 

0.15 

0.440382308 

0.664871044 

0.66403832 

0.15168111 

0.439719228 

103.0336222 

107 

96 

-0.2 

0.579259709" 

0.668700721 

0.678896435 

-0.217222651 

0.585982583 

95.65554698 

108 

92 

-0.4 

0.655421742 

0.65279443 

0.642282328 

-0.383053994 

0.649160135 

92.33892013 

112 

103 

0.15 

0.440382308 

0.689781704 

0.687766474 

0.153926677 

0.438833775 

103.0785335 

117” 

. 88 

-0.6 

0.725746882 

0.686601595 

0.688882708 

-0.603129218 

0.726788635 

87.93741565 

118 

88 

-0.6 

0.725746882 

0.707626076 

0.675890021 

-0.556215773 

0.710968321 

88.87568455 

”  119 

85” 

-0.75 

0.773372648 

0.658118465 

0.674468889 

-0.771426019 

0.779772771 

84.57147962 

120 

92 

-0.4 

0.655421742 

0.666742561 

0.688851107 

-0.433641766 

0.6677257 

91.32716468 

121 

88 

-0.6 

0.725746882 

0.733410493 

0.701855772 

-0.55805483 

0.711596528 

88.83890339 

”  '122 

85 

-0.75 

0.773372648 

0.709052039 

0.709052039 

-0.75 

0.773372648 

85 

~  123 

97 

-0.15 

0.559617692 

0.650034124 

0.645589902 

-0.141960302 

0.556444319 

97.16079395 

124 

107 

0.35 

0.363169349 

0.723223681 

0.676987953 

0.448751292 

0.32680554" 

108.9750258 

125 

92 

-0.4 

0.655421742 

0.723462278 

0.688546032 

-0.347932303 

0.636054488 

93.04135393 

”'126 

92 

-0.4 

0.655421742 

0.731579645 

0.692634942 

-0.342343802” 

0.633953912“ 

93.15312396 

129 

107 

0.35 

0.363169349 

0.713964828 

0.676729247 

0.429741524 

0.333691837 

108.5948305 

”130 

95 

-0.25 

0.598706326 

0.703489738 

0.703489738 

-0.25 

0.598706326 

95 

131 

”  107 

0.35 

0.363169349 

0.740458173 

0.701597213 

0.430267425 

0.33350056 

108.6053485 

'■'132 

92 

-0.4 

0.655421742 

0.661641669 

0.627934561 

-0.344922685 

0.634923752 

rt,  1  1  •  _ 

93.10154631 

/~\1  J  A  A 

Note.  MU5  =  MUi  UJ  iNumoer,  jr^o  -  wiu  ^  ji  - - - - 

Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  =  Validity  of  9  JF-Level  AA  Composite,  R1 
=  Validity  of  1 7  JF  Level  AA  Composite;  ZC1  =  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF17CS  =  New  AA  Cut  Score  (17  JF  Level). 
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Table  1  (cont’d) 

New  and  Old  Cut  Scores  and  Qualification  Rates  by  MOS  for  1 7  Job  Families 


MOS 

JF9CS 

ZCO 

Q0 

R0 

R1 

ZC1 

Q1 

JF17CS 

133 

96 

-0.2 

0.579259709 

0.703459776 

0.668330344 

-0.140644133 

0.555924459 

97.18711734 

134 

102 

0.1 

0.460172163 

0.540006869 

0.505735404 

0.187672118 

0.425566844 

103.7534424 

135 

92 

-0.4 

0.655421742 

0.678738208 

0.639929164 

-0.33787774 

0.632272338 

93.24244521 

137 

92 

-0.4 

0.655421742 

0.723581145 

0.715971111 

-0.38897919 

0.65135423 

92.22041621 

138 

85 

-0.75 

0.773372648 

0.697797652 

0.697797652 

-0.75 

0.773372648 

85 

139 

88' 

-0.6 

0.725746882 

0.674647315 

0.64509594 

-0.557270966 

0.711328845 

88.85458069 

140 

88 

-0.6 

0.725746882 

0.667530862 

0.639479244 

-0.559062609 

0.711940504 

88.81874782 

141 

92 

-0.4 

0.655421742 

0.721646975 

0.706723333 

-0.378161655 

0.647344746 

92.43676689 

142 

96 

-0.2 

0.579259709 

0.737485532 

0.757922758 

-0.231011139 

0.591346926 

95.37977722 

143 

93 

-0.35 

0.636830651 

0.721584522 

0.718489973 

-0.345419177 

0.635110369 

93.09161645 

144 

92 

-0.4 

0.655421742 

0.704051744 

0.722670289 

-0.42696104 

0.665296148 

91.46077919 

145 

92 

-0.4 

0.655421742 

0.68449826 

0.702443632 

-0.426733006 

0.665213096 

91 .46533987 

146 

96 

-0.2 

0.579259709 

0.696044518 

0.715326945 

-0.231001086 

0.591343021 

95.37997827 

147 

102 

0.1 

0.460172163 

0.633256782 

0.6489369 

0.068155869 

0.472830779 

101.3631174 

148 

102 

0.1 

0.460172163 

0.63754039 

0.658876113 

0.057249181 

0.477173351 

101.1449836 

149 

93 

-0.35 

0.636830651 

0.657476775 

0.658242868 

-0.351239503 

0.637295662 

92.97520993 

150 

102 

0.1 

0.460172163 

0.697719236 

0.725191857 

0.049927471 

0.480090093 

100.9985494 

151 

92 

-0.4 

0.655421742 

0.713548551 

0.740726056 

-0.438504926 

0.66948985 

91 .22990148 

152 

102 

0.1 

|  0.460172163 

0.704493117 

0.729707014 

0.054361242 

0.478323679 

101 .0872248" 

154 

92 

-0.4 

]  0.655421742 

0.691700773 

0.717427129 

-0.437624481 

0.669170739 

91.24751038 

Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  =  Validity  of  9  JF-Level  AA  Composite;  R1 
=  Validity  of  17  JF  Level  AA  Composite;  ZC1  =  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF17CS  =  New  AA  Cut  Score  (17  JF  Level). 
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Table  2 

New  and  Old  Cut  Scores  and  Qualification  Rates  by  MOSfor  ISO  Job  Families 


MOS 

JF9CS 

ZCO 

Q0  ; 

R0 

R1 

ZC1 

Q1 

JF150CS 

1 

87 

-0.65 

0.7421 53889 1 

0.528374924 

0.4752681 

-0.549999321 

0.70884008 

89.00001359 

2 

87 

-0.65 

0.742153889 

0.602825356 

0.59534875 

-0.638467207 

0.738415203 

87.23065586 

3 

87 

-0.65 

0.7421 538891 

0.5907829091 

0.582748295 

-0.63734273 

0.738049188 

87.2531454 

4 

87 

-0.65 

0.742153889 

0.540846301 

0.49605518 

-0.568756024 

0.715239139 

88.62487953 

5 

87 

-0.65 

0.742153889 

0.593138316 

0.583612345 

-0.635025874 

0.73729423 

87.29948252 

. 6 

87 

-0.65 

0.742153889 

0.59534116 

0.593876042 

-0.647728155 

0.741419605 

87.0454369 

8 

91 

-0.45 

0.67364478 

0.624690167 

0.583837466 

-0.3801 96228 1 

0.648100121 

92.39607544 

"9 

91 

-0.45 

0.67364478 

0.569657011 

0.569932557 

-0.450490801 

0.673821707 

90.99018398 

"To 

Q1 

-0.45 

0.67364478 

0.66029159 

0.64923422TI 

-0.432787255 

0.667415333 

91 .34425489 

ii 

91 

-0.45 

0.67364478 

0.662381536 

0.653318653 

-0.435969035 

0.668570402 

91.2806193 

12 

95 

-0251 

0.598706326 

0.6633241 051 

0.60326404 

-0.141336758 

0.556198044 

97.17326484 

14 

98 

-0.1 

0.539827837 

0.6486804151 

0.580543473 

0.037074543 

0.485212785 

100.7414909 

22 

87 

-0.65 

0.742153889 

0.605869684 

0.608148781 

-0.653457006 

0.743269151 

86.93085989 

24 

87 

-0.65 

0.742153889 

0.620224189 

0.634552351 

-0.670939019 

0.748870311 

86.58121961 

26 

92 

-0.4 

0.655421742 

0.689912396 

0.613927381 

-0.275045676 

0.608359427 

94.49908647 

“  29 

"  102 

0.1 

0.4601 721 631 

0.5921857 

0.497792982 

0.340002407 

0.366927357 

106.8000481 

. 33 

98 

-0.1 

0.539827837 

0.669540892 

0.617647499 

-0.001212665 

0.500483783 

99.97574669" 

35 

89 

-0.55 

0.708840313 

0.610008827 

0.547886292 

-0.443413884 

0.671266802 

91.13172231 

37 

107 

0.35 

0.363169349 

0.701824913 

0.582664361 

0.639346246 

0.261298855 

112.7869249 

39 

98 

-0.1 

0.539827837 

0.62863297 

0.589447296 

-0.021550936 

0.508596914 

99.56898129 

40 

116 

0.8 

0.211855399 

0.665774129 

0.562044096 

1.117181308 

0.131958405 

122.3436262 

42 

102 

0.1 

0.460172163 

0.667514629 

0.600896509 

0.242279127 

0.404281943 

104.8455825 

~  43 

107 

0.35 

0.363169349 

0.668178174 

0.568036453 

0.600490253 

0.274089777 

112.0098051 

'44 

102 

0.1 

0.460172163 

0.685673394 

0.586686166 

0.314314816 

0.376640973 

106.2862963 

45 

98 

-0.1 

0.539827837 

0.647524571 

0.570754226 

0.056562875 

0.477446704 

101.1312575 

48 

88 

-0.6 

0.725746882 

0.72296745 

0.667234012 

-0.522832647 

0.699454641 

89.54334707 

49 

97 

-0.15 

0.559617692 

0.735289835 

0.672379244 

-0.042761147 

0.517054032 

99.14477707 

. . 50 

88 

-0.6 

0.725746882 

0.600596715 

0.563205817 

-0.538401086 

0.704849912 

89.23197828 

51 

93 

-0.35 

0.636830651 

0.640425362 

0.546217486 

-0.173423593 

0.568840759 

96.53152813 

52 

97 

-0.15 

0.559617692 

0.694022416 

0.570228887 

0.092882317 

0.462998527 

101.8576463" 

53 

97 

-0.15 

0.559617692 

0.71979739 

0.661440112 

-0.048765357 

0.519446855 

99.02469287 

"  56 

93 

-0.35 

0.636830651 

0.735260276 

0.660442092 

-0.232534139 

0.591938413 

95.34931722 

58 

'  "  88 

-0.6 

0.725746882 

0.701389192 

0.625608476 

-0.489125181 

0.687623462 

90.21749639 

59 

88 

-0.6 

0.725746882 

0.686933742 

0.620497789 

-0.50166076 

0.692046914 

89.9667848 

60 

'  88 

-0.6 

0.725746882 

0.656507343 

0.543248141 

-0.41301169 

0.660200975 

91 .73976621 

"61 

93 

-0.35 

0.636830651 

0.728627437 

0.6627156 

-0.246554585 

0.597373522 

95.0689083 

62 

'  102 

0.1 

0.460172163' 

0.60610754 

0.545345245 

0.242977269 

0.404011503 

1 04.8595454 

63 

. 97 

-0.15 

0.559617692 

0.680025098' 

0.60253374 

-0.003643765 

0.501453649 

99.9271247 

64 

97 

-0.15 

0.559617692 

0.709182712 

0.647485687 

-0.04082538 

0.516282447 

99.1834924 

65 

92 

-0.4 

0.655421742 

0.746809578 

0.71585551 

-0.355522 

0.638900717 

92.88955999 

. 66 

96 

-0.2 

0.579259709  0.664528937 

0.60542832 

-0.090832655 

0.536187219 

98.18334689 

67 

104 

0.2 

0.420740291 

0.724644707 

0.637454212 

0.3828734 

0.350906817 

107.657468 

70 

87 

-0.65 

0.742153889' 

0.70000591 

0.656400152 

-0.589860491 

0.722357908 

88.20279019 

-  jy 

88 

-0.6 

0.725746882 

0.7401448641  0.665260459 

-0.496752332 

0.690318143 

1  J! _ 1 

90.06495335 

r\i  J  a  a 

Note.  MUo  —  MUo  lL;  iNUIUDCI,  —  WIU  V^ui  jwiv  ~~~  - - - - 

Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  =  Validity  of  9  JF-Level  AA  Composite,  R1 
=  Validity  of  150  JF  Level  AA  Composite;  ZC1  =  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF150CS  =  New  AA  Cut  Score  (150  JF  Level). 
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Table  2  (cont’d) 

New  and  Old  Cut  Scores  and  Qualification  Rates  by  MOS  for  150  Job  Families 


MOS| 

JF9CS 

ZCO 

Q0 

R0  ! 

R1 

ZC1 

Q1 

JF150CS 

72 

88 

-0.6 

0.725746882 

0.705386736, 

0.640135999 

-0.506259999 

0.69366293 

89.87480002 

73 

88 

-0.6 

0.725746882 

0.641538019! 

0.585418578 

-0.511710424 

0.695573155 

89.76579151 

74 

87 

-0.65 

0.742153889 

0.713812452 

0.664743539 

-0.583302578 

0.720155186 

88.33394843 

75' 

97 

-0.15 

0.559617692 

0.764372563 

0.655285599 

0.038037174 

0.484829021 

100.7607435 

76 

97 

-0.15 

0.559617692 

0.792072668 

0.702993657 

-0.005745797 

0.502292229 

99.88508405 

77 

97 

-0.15 

0.559617692 

0.7505519961 

0.638492143 

0.047897481 

0.480898974 

100.9579496 

78 

87 

-0.65 

0.742153889 

0.676698726 

0.577052888 

-0.497725101 

0.690661093 

90.04549799 

79] 

87 

-0.65 

0.742153889 

0.64366095 

0.554044258 

-0.506994237 

0.693920569 

89.86011526 

81 

97 

-0.15 

0.559617692 

0.728881979 

0.62497767 

0.037797144 

0.484924711 

100.7559429 

82 1 

102 

0.1 

0.460172163 

0.69485919 

0.586903212 

0.333037117 

0.369553133 

106.6607423 

83 

87 

-0.65 

0.742153889 

0.750439786 

0.705636543 

-0.592476656 

0.723234276 

88.15046688 

84 

97 

-0.15 

0.559617692 

0.691086211 

0.601728946 

0.018326624 

0.492689144 

100.3665325 

85 

102 

0.1 

0.460172163 

0.701452311 

0.631130181 

0.242980631 

0.404010201 

104.8596126 

86 

97 

-0.15 

0.559617692 

0.70697104 

0.61064814 

0.0284746781 

0.488641782 

100.5694936 

87 

102 

0.1 

0.460172163 

0.717587198 

0.619901888 

0.300551548 

0.381878242 

106.011031 

88 

102 

0.1 

0.460172163 

0.627235496 

0.538203859 

0.310244414 

0.378187549 

106.2048883 

89 

102 

0.1 

0.460172163 

0.711505261 

0.599025674 

0.33773355 

0.367781995 

106.754671 

90 

102 

0.1 

0.460172163 

0.672302894 

0.578822271 

0.305399418 

0.380030982 

106.1079884 

91 

102 

0.1 

0.460172163 

0.738440547 

0.520344156 

0.612226785 

0.270193861 

112.2445357 

92 1 

102 

0.1 

0.460172163 

0.7843001 071 

0.649859209 

0.361083378 

0.359018559 

107.2216676 

93 

102 

0.1 

0.460172163 

0.784078298 

0.674131718 

0.307366949 

0.379282041 

106.147339 

94 

102 

0.1 

0.460172163 

0.71409059 

0.650047849 

0.226724563 

0.410318963 

104.5344913 

95 

93 

-0.35 

0.636830651 

0.721649898 

0.636957623 

-0.212721965 

0.584228083' 

95.7455607 

97 

93 

-0.35 

0.636830651 

0.637569126 

0.587380383 

-0.260847518 

0.602894951 

94.78304965 

99 

107' 

0.35 

0.363169349 

0.423410229 

0.356023067 

0.618407143 

0.268153495 

112.3681429 

101 

92 

-0.4 

0.655421742 

0.698706419 

0.59428135 

-0.224598824 

0.58885431 

95.50802353 

105 

92 

-0.4 

0.655421742 

0.65003742 

0.551062966 

-0.220864184 

0.587400907 

95.58271633 

106 

103 

0.15 

0.440382308 

0.664871044 

0.577736298 

0.346638438 

0.364431483 

106.9327688 

107 

96 

-0.2 

0.579259709 

0.668700721 

0.567639178 

-0.004122272 

0.501644544 

99.91755457 

108 

92 

-0.4 

0.655421742 

0.65279443 

0.576894781 

-0.267404109' 

0.605420982 

94.65191783 

112 

_ 103 

0.15 

0.440382308 

0.689781704 

0.585243736 

0.381790438 

0.351308406 

107.6358088 

117 

88 

-0.6 

0.725746882 

0.686601595 

0.686187842 

-0.599430793 

0.725557176 

88.01138415 

118 

88 

-0.6 

0.725746882 

0.707626076 

0.609817015 

-0.454554038 

0.675284946 

90.90891924 

119 

85 

-0.75 

0.773372648 

0.658118465' 

0.529017661 

-0.548807564 

0.708431241 

89.02384872 

120 

92 

-0.4 

0.655421742 

0.666742561 

0.629788933 

-0.339867233 

0.633021743 

93.20265535 

121 

88 

-0.6 

0.725746882 

0.733410493 

0.603876116 

-0.407871695 

0.658316063 

91.8425661 

122 

85 

-0.75 

0.773372648 

0.709052039 

0.662762631 

-0.689873342 

0.754863079 

86.20253316 

“123 

97 

-0.15 

0.559617692 

0.650034124 

0.503509664 

0.171302032 

0.431993142 

103.4260406 

124 

“'107 

0.35 

0.363169349 

0.723223681 

0.636415672 

0.545018727 

0.292870319 

110.9003745 

125 

92 

-0.4 

0.655421742 

0.723462278 

0.634518572 

-0.258995779 

0.602180751 

94.82008442 

126 

92 

-0.4 

0.655421742 

0.731579645 

0.588233432 

-0.160144163 

0.563616244 

96.79711674 

129 

107 

0.35 

0.363169349 

0.713964828 

0.506940659 

0.912092839 

0.180859924 

118.2418568 

“130 

95 

-0.25 

0.598706326 

0.703489738 

0.618740187 

-0.101679041 

0.540494281 

97.96641918 

131 

107 

0.35 

0.363169349 

0.740458173 

0.657309146 

0.531148154 

0.297658059 

110.6229631 

132 

92 

-0.4 

0.655421742 

0.661641669 

0.602385987 

-0.300117834 

0.617956362 

93.99764332 

Note.  MOS  =  MOS  ID  Number;  JF9CS  =  Old  AA  Cut  Score  (9  JF  Level);  ZCO  =  Standardized  Old  AA 
Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  =  Validity  of  9  JF-Level  AA  Composite;  R1 
=  Validity  of  150  JF  Level  AA  Composite;  ZC1  =  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF150CS  =  New  AA  Cut  Score  (150  JF  Level). 
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Table  2  (cont’d) 

New  and  Old  Cut  Scores  and  Qualification  Rates  by  MOS  for  150  Job  Families 


MOS 

JF9CS 

ZCO 

Q0 

R0 

R1 

ZC1  ] 

Q1 

JF150CS 

133 

96 

-0.2, 

0.5792597091 

0.703459776 

0.639431045 

-0.088078269 

0.535092766 

98.23843463 

134 

“102 

”  0.1 

0.460172163 

0.540006869 

0.467668426 

0.296957005 

0.383249667 

105.9391401 

135 

92 

-0.4 

0.655421742 

0.678738208 

0.591156765 

-0.251231385 

0.59918239 

94.9753723 

137 

. 92 

-0.4 

0.655421742 

0.723581145' 

0.65403854 

-0.292231207 

0.614945073 

94.15537587 

138 

85 

-0.75 

0.773372648 

0.697797652 

0.667701054 

-0.71093473 

0.761437658 

85.7813054 

139 

88 

-0.6 

0.725746882 

0.674647315 

0.603321784 

-0.491712326 

0.688538638 

90.16575348 

140 

88 

-0.6 

0.7257468821 

0.6675308621 

0.532245086 

-0.374310745 

0.645913431 

92.5137851 

141 

92 

-0.4 

0.655421742 

0.721646975 

0.621424046 

-0.238510865 

0.594257555 

95.2297827 

142 

96 

-0.2 

0.579259709 

0.737485532 

0.6514227481 

-0.05330761 

0.521256592 

98.9338478 

'143 

93 

-0.35 

0.636830651 

0.7215845221 

0.652192158^ 

-0.239506072 

0.594643406 

95.20987857 

144 

92 

-0.4 

0.655421742 

0.704051744“ 

0.650429566 

-0.315982375 

0.623992057 

93.68035251 

"  145" 

92 

-0.4 

0.655421742 

0.68449826 

0.612574471 

-0.281294117 

0.610757589 

94.37411766 

"146 

96 

-0.2 

0.579259709 

0.696044518 

0.596800715 

-0.016621562 

0.506630739 

99.66756876 

147 

102 

0.1 

0.460172163 

0.633256782 

0.541478552 

0.315267912 

0.376279123 

106.3053582 

“148 

102 

0.1 

0.460172163 

0.63754039 

0.590475826 

0.202886347 

0.419611933 

104.0577269 

149” 

93 

-0.35 

0.636830651 

0.657476775 

0.604700706 

-0.258974552 

0.602172562 

94.82050897 

150 

'102 

0.1 

0.460172163 

0.697719236 

0.619541292 

0.261492955 

0.396856191 

105.2298591 

151 

. . 92 

-0.4 

0.655421742 

0.713548551 

0.629789208 

-0.266005378 

0.604882467 

94.67989243 

152 

102 

0.1 

0.460172163 

0.704493117" 

0.617336211 

0.280199843 

0.389662094 

105.6039969 

154 

. . ”92 

-0.4 

0.655421742 

0.691700773 

0.629281011 

-0.299300219 

0.617644506 

94.01399561 

Note. 

MOS  = 

mos: 

ID  Number;  JF9CS  =  Old  AA  Cut  Score  (9  Jl 

Level);  ZCO  = 

Standardized  Old  AA 

Cut  Score;  QO  =  Qualification  Rate  Under  Old  Cut  Score;  RO  —  Validity  of  9  JF -Level  AA  Composite,  R1 
=  Validity  of  150  JF  Level  AA  Composite;  ZC1  .=  Standardized  New  AA  Cut  Score;  Q1  =  Qualification 
Rate  Using  New  Cut  Score;  JF150CS  =  New  AA  Cut  Score  (150  JF  Level). 
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